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Preface 


The  development  of  a  routing  algorithm  for  the  new 
Automatic  Digital  Netwrok  II  (AUTODIN  II)  has  been  a  topic 
of  major  concern  for  two  years.  The  Defense  Communications 
Agency  (DCA)  contracted  out  to  Western  Union  Telegraph 
Company  the  design  and  development  of  the  AUTODIN  II  net¬ 
work  configuration  and  systems  engineering.  As  authorized 
through  Western  Union  by  DCA,  Systems  Control  Incorporated 
(SCI)  was  contracted  to  develop  a  routing  algorithm  based 
on  the  Advanced  Research  Projects  Agency  Network  (ARPANET) 
routing  algorithm  but  tailored  specifically  to  the 
AUTODIN  II  network. 

In  essence,  this  research  project  is  a  computer  sim¬ 
ulation  of  the  SCI  network  routing  algorithm  to  determine 
its  characteristics  and  limitations  under  varying  degrees 
of  network  configuration  and  system  stress  conditions. 

I  would  like  to  express  my  thanks  to  Capt  Wade  L. 
Neilson  for  his  help  in  gathering  the  technical  data  on  the 
algorithm.  In  addition,  I  want  to  express  my  sincere  appre¬ 
ciation  to  Dr.  Tom  Hartrum  and  Capt  Larry  Kizer  whose  tech¬ 
nical  expertise  proved  invaluable.  A  very  special  thanks 
goes  to  ay  thesis  advisor,  Major  Allen  A.  Ross,  for  his 
constructive  advice,  technical  assistance  and  a  sincere 
interest  in  the  project  from  its  conception  to  completion. 

My  deepest  appreciation  goes  to  my  loving  wife,  Lorraine, 
whose  moral  support  and  sympathetic  understanding  indured 

me  to  continue  throughout  the  course  of  the  project. 

John  A.  Whit ten ton 
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Abstract 


In  computer  communications  systems,  a  network  routing 
algorithm  is  the  methodology  behind  which  a  message  is  di¬ 
rected  (routed)  from  a  source  computer  site,  through  the 
network  of  linked  computers,  to  a  destination  computer  site. 

The  simulation  of  such  an  algorithm  is  of  extreme 
importance  in  determining  the  efficiency  of  the  algorithm, 
its  reaction  response  to  network  topology  changes,  and  its 
system  response  as  the  network  capacity  of  messages  reaches 
the  saturation  point. 

The  algorithm  to  be  simulated  is  hierarchical  in  nature, 
that  is,  with  multiple  processors  at  each  computer  site 
(node),  the  best  route  to  the  succeeding  node  is  selected 
first,  then  a  processor  having  a  physical  communications 
line  (trunk)  in  the  best  route  is  selected.  The  connectivity 
of  the  network  (how  the  nodes  are  connected  to  one  another) 
and  the  amount  of  congestion  at  each  node  are  vital  parameters 
used  by  the  algorithm  in  determining  the  best  route. 

The  simulation  language  selected  was  General  Purpose 
Simulation  System  (GPSS).  It  is  an  event  oriented  language 
readily  adaptable  to  network  queueing  models.  Entities 
called  transactions  are  generated  to  simulate  message  seg¬ 
ments  (packets)  which  are  then  processed  as  a  normal  packet 
would  be  in  a  communications  processor.  Time  delays  are 
incurred  by  the  transaction  due  to  processing  of  the  packet, 
wait  time  in  queues  and  transmission  delays  (the  time  it 
takes  to  transmit  the  packet  from  one  node  to  another). 
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The  actual  algorithm  is  written  in  FORTRAN.  GPSS 
instruction  blocks  are  used  to  jump  from  the  GPSS  code 
to  the  FORTRAN  subroutines  where  the  actual  path  is  cal¬ 
culated  that  the  packet  will  take  to  the  next  node. 

A  baseline  program  with  static  routing,  that  is, 
fixed  routing  that  does  not  change,  was  first  developed 
to  establish  a  control  program. 

Four  levels  of  input  packet  generation  were  used  to 
vary  the  network  loading  from  60#  to  90#  of  network  trunk 
saturation.  In  addition  to  varying  the  traffic  volume 
through  the  network,  selected  transmission  trunks  and  links, 
and  nodes  were  removed  from  the  network  to  simulate  varying 
topology  changes. 
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AN  ANALYSIS  OP  A  ROUTING  ALGORITHM 


POR  AUTODIN  II 
I  Introduction 


Background 

A  new  computerized  defense  communications  network, 
Automated  Digital  Network  II  (AUTODIN  II),  is  currently 
being  designed  by  the  Defense  Communications  Agency  (DCA). 
The  new  system  is  a  result  of  increased  user  requirements 
on  the  current  AUTODIN  I  network,  additional  independent 
military  Automated  Data  Processing  (ADP)  systems  requiring 
computer  communications  support,  and  projected  computer- 
oriented  military  information  systems  requesting  entry  into 
the  World  Wide  Military  Command  and  Control  System  (WWMCCS) 
Standard  ADP  System  (Ref  1>1-1).  In  1972,  DCA  began  con¬ 
ducting  a  comprehensive  study  to  identify  current  and  fu¬ 
ture  ADP  requirements.  The  study  concluded  that  by  19?6 
there  would  be  approximately  250  computers  involved  in 
on-line  communications  functions  with  approximately  8,000 
input/output  terminals  (Ref  lil-l)«  Projections  into  the 
mid  1980*8  indicated  that  within  the  Department  of  Defense 
(DoD)  there  will  be  approximately  2,500  computers  involved 
in  communications  with  some  20,000  input/output  terminals 
(Ref  lil-l).  The  approach  to  AUTODIN  II  is  start  small 
(8  computer  switching  centers),  meet  the  known  validated 
requirements,  and  grow/evolve  as  requirements  increase  and 
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the  needs  for  service  change 


In  June  1975  the  Director,  Telecommunications  and 
Command  and  Control  Systems  (DTACCS),  Office  of  the  Sec¬ 
retary  of  Defense,  gave  initial  program  approval  for 
AUTODIN  II 's  development.  The  Request  for  Proposals  was 
released  to  industry  on  25  November  1975*  with  proposals 
due  by  7  April  1976 j  final  program  approval  was  obtained 
in  November  1976  with  subsequent  contract  award  to  the 
Western  Union  Telegraph  Company  and  its  major  subcontrac¬ 
tors,  Ford  Aerospace  and  Communications  Corporation  and 
Computer  Services  Corporation  (Ref  lil-2).  The  Govern¬ 
ment  issued  the  order  to  proceed  with  design  and  implemen¬ 
tation  of  an  initial  four  computer  network  on  14  January  1977 
(Ref  lil-2) • 

AUTODIN  II  is  a  digital,  common-user,  distributed 
communications  computer  network  employing  a  message  trans¬ 
mittal  methodology  called  message  packet-switching.  A 
message  is  segmented  into  finite  length  segments  called 
packets.  Each  packet  contains  all  the  essential  infor¬ 
mation  necessary  for  routing  processes  in  addition  to  the 
actual  data  to  be  transmitted.  The  procedures  or  processes 
which  direct  (route)  a  message  or  packet  from  a  source 
computer  site  through  the  network  of  linked  computers,  to 
a  destination  computer  site  are  called  routing  algorithms. 

Purpose/Problem 

A  routing  algorithm  has  been  developed  by  Systems 
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Control  Incorporated  similar  to  the  one  used  by  the  Ad¬ 
vanced  Research  Projects  Agency  Network  (ARPANET)  but 
with  added  refinements  to  tune  it  to  the  AUTCDIN  II  net¬ 
work.  An  algorithm  developed  for  use  in  a  highly  sophis¬ 
ticated  and  complex  network  should  be  able  to  handle  var¬ 
iations  (spiked  increases)  in  normal  message  volume  with 
packets  accumulating  only  minor  additional  delay  time.  It 
should  also  be  able  to  detect  any  topological  change  or 
congestion  occurring  in  the  network  and  reroute  the  pac¬ 
ket  accordingly#  with  minimum  additional  delay  in  packet 
delivery.  The  purpose  of  this  thesis  is  to  determine  the 
behavior  characteristics  of  the  routing  algorithm  under 
various  simulated  conditions  involving  the  introduction 
of  spikes  in  message  volume  and  changes  in  network  topology. 


Scope 

* 

The  content  of  this  report  is  limited  exclusively  to 
the  description,  incorporation  of  the  algorithm  into  the 
simulation  model,  and  execution  of  the  AUTODIN  II  routing 
algorithm.  Detailed  technical  descriptions  of  other  net¬ 
work  system  components  are  beyond  the  scope  of  this  paper. 

A  network  model  is  developed  with  a  static  (non¬ 
changing,  deterministic)  routing  table  as  a  control  model. 
The  algorithm  will  then  be  incorporated  into  the  network 
simulation  model. 


This  report  contains  nine  chapters.  Chapter  II  begins 
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with  a  brief  explanation  of  computer  networks  in  general, 
then  proceeds  with  a  description  of  the  AUTODIN  II  network 
architecture.  Network  topology,  packet-switch  center  (node) 
decomposition,  and  network  connectivity  are  also  discussed. 
Chapter  III  describes  the  types  of  routing  algorithms  used 
in  various  computer  networks.  Chapter  IV  describes  the 
developed  hierarchical  routing  algorithm  in  detail.  The 
six  modules  composing  the  algorithm  and  their  interrelation¬ 
ships  are  discussed.  The  two  modes  of  the  algorithm,  "Per 
Packet  Functions"  and  "Background  Functions"  are  also  covered. 

Chapter  V  discusses  the  simulation  language,  General 
Purpose  Simulation  System  (GPSS),  and  the  network  simulation 
model.  In  Chapter  VI,  the  baseline  simulation  model  is 
developed  with  discussions  on  the  static  routing  table,  pac¬ 
ket  generation,  source  node  traffic  routing  matrix  and  other 
essential  baseline  considerations. 

Chapter  VII  deals  with  how  the  algorithm  is  incorporated 
into  the  network  simulation  model.  Traffic  volume  changes 
and  topological  changes  are  also  explained. 

Chapter  VIII  discusses  the  results  of  the  simulation 
with  emphasis  on  packet  delay  analysis.  In  Chapter  IX  con¬ 
clusions  on  the  algorithm *8  adaptability  to  network  changes 
and  congestion,  and  any  abnormal  results  are  stated.  Recom¬ 
mendations  on  possible  changes  or  modifications  to  the  al¬ 
gorithm  or  its  documentation  and  any  advisable  network  re¬ 
configurations  are  also  included  in  Chapter  IX. 


Basic  Computer  Networks 


In  general,  a  computer  network  is  formed  from  the  tying 
or  linking  together  of  two  or  more  computers  with  their  asso¬ 
ciated  communications  interfaces.  The  computers  that  process 
and  direct  messages  or  packets  along  to  their  destinations 
are  called  switching  computers  or  nodes.  The  locations  of 
the  nodes  can  be  geographically  dispersed  (that  is,  a  site 
on  the  East  Coast  and  a  site  on  the  West  Coast),  or  locally 
connected  (that  is,  intercity).  Connecting  the  nodes  togeth¬ 
er  are  physical  data  communications  links  called  trunks. 

The  trunks  are  made  up  of  leased  land  lines,  microwave  links, 
satellite  channels,  or  any  combination  of  these. 

Host  computers  and/or  terminals  are  usually  connected 
to  the  nodes.  The  network  services  the  host  computers  and 
terminals  by  directing  or  routing  the  data  messages  to  and 
from  other  host  computers  and  terminals  either  connected  to 
the  same  node  or  located  at  some  other  distant  node.  The 
primary  purpose  of  the  network  is  to  permit  access  by  a 
user,  or  by  a  process  acting  on  behalf  of  the  user,  to  re¬ 
sources  of  the  host  computers  (Ref  2i48).  The  primary  func¬ 
tion  of  the  network  is  to  provide  the  medium  through  which 
the  user* s  data  messages  are  input  from  his  equipment,  pro¬ 
perly  routed  to  their  destination  node,  and  output  to  the 
destination's  equipment.  The  network  should  be  invisible 
to  the  users,  providing  only  a  virtual  connection  between 
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the  user's  input  equipment  and  the  Host  computer  the  user 
is  querying. 


Network  Description 

The  proposed  AUTODIN  II  network  is  the  result  of 
several  comprehensive  communications  computer  requirements 
surveys  (Ref  1«1-1).  There  are  numerous  existing  and  near- 
future  computer  systems  employing  various  techniques  for 
common-user  communications  between  various  military  organ¬ 
izations.  The  surveys  consolidated  the  different  organiza¬ 
tional  requirements,  near  future  and  tentative  computer 
systems  requirements  for  communications  computer  support. 

The  end  result  was  the  AUTODIN  II  network  (Ref  3). 

AUTODIN  II  is  designed  to  provide  economical  and  relia¬ 
ble  data  communications  service  both  for  interactive  time¬ 
sharing  and  transaction-oriented  systems  requiring  rapid 
response  between  terminals  and  computer-to-computer  data 
transfers  requiring  high  transmission  capacity  (Ref  3)» 

Prom  the  source  or  initial  node,  each  packet  is  dynamically 
routed  along  one  of  several  alternate  paths,  in  some  cases 
passing  through  one  or  more  tandem  nodes  before  arriving  at 
the  destination  node.  Unlike  AUTODIN  I  which  requires  a 
message  to  be  received  in  its  entirety  at  a  node  before 
transmitting  the  message  further,  AUTODIN  II  provides  the 
capability  of  routing  simultaneously  packets  (containing 
part  of  the  original  message)  independently  of  other  packets 
of  the  same  message  thus  propagating  the  message  through  the 
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network  faster.  If  a  message  is  made  up  of  multiple 
packets,  each  packet  can  be  routed  independently  of  the 
other  packets.  Thus  several  packets  can  be  propagated 
through  the  network  at  the  same  time,  not  necessarily 
using  the  same  route  to  get  to  the  destination  node.  If 
there  are  two  separate  paths  to  get  to  the  destination, 
half  the  packets  can  be  transmitted  on  one  path,  the  other 
half  being  transmitted  on  the  other  path.  In  effect,  the 
packets  parallel  each  other  in  arriving  at  their  destination 
node.  A  packet  incurs  a  delay  of  only  a  very  small  fraction 
of  a  second  at  each  node,  so  that  the  total  delay,  from  the 
time  a  packet  fully  enters  the  initial  node  from  the  user 
(subscriber)  until  it  begins  to  leave  the  destination  node, 
shall  be  less  than  a  second  (Ref  3). 

Network  Architecture 

Topologv/Connectivitv.  The  network  is  currently  made 
up  of  eight  packet-switching  nodes  (PSNs)  located  at  Albany, 
GAt  Andrews  AFB,  MDj  Ft.  Detrick,  MD|  Gentile  AFS,  0H»  Han¬ 
cock  Field,  NYi  McClellan  AFB,  CAj  Norton  AFB,  CAj  and  Tin- 
AFB,  OK  as  Figure  1  depicts.  The  numbers  in  the  figure 
represent  how  many  trunks  are  contained  in  each  link. 

These  also  are  the  sites  of  the  current  AUTODIN  I  network. 
The  decision  was  made  to  colocate  the  two  networks  for  ease 
of  maintainability,  cutover  procedures,  and  obvious  budget 
considerations . 

The  connectivity  of  the  network  is  an  important  factor 


7 


Fig  1.  The  AUTODIN  II  Network 


in  the  network.  If  each  node  in  the  network  has  three 
other  nodes  connected  to  it,  then  the  connectivity  of  the 
network  is  defined  to  he  3*0.  Unlike  the  ARPANET  with 
connectivity  2.4  (2.4  nodes  connect  to  each  node),  there 
are  an  average  of  four  other  nodes  connecting  each  node 
in  the  AUTODIN  II  network  representing  a  connectivity  of 
4.0.  Packet  wise,  this  means  that  there  are  fewer  nodes, 
and  therefore  fewer  links,  for  a  packet  to  traverse  in 
order  to  arrive  at  its  destination  node.  The  average  num¬ 
ber  of  hops  (link  traversals)  a  packet  takes  in  the 
AUTODIN  II  network  is  .84,  whereas  in  ARPANET  it  is  6  (Ref  3). 
The  result  of  high  network  connectivity  is  less  time  delay 
from  packet  entry  into  the  source  node  to  delivery  at  the 
destination  node.  High  network  connectivity,  as  in  the 
AUTODIN  II  network,  means  a  packet  requires  fewer  hops  to 
get  to  its  destination.  With  each  hop,  a  packet  incurs  de¬ 
lay  by  having  to  be  input  processed  by  the  receiving  node 
and  also  by  having  to  be  retransmitted  to  the  next  node. 

The  fewer  hops  a  packet  must  make,  the  smaller  the  delay  is 
at  arriving  at  its  destination. 

Node  Decomposition.  Each  packet-switching  node  (PSN) 
contains  from  one  to  three  PDP  11/70  processors  (Switch  Con¬ 
trol  Modules  (SCMs)),  depending  on  the  particular  PSN's 
average  traffic  volume.  Interconnecting  the  nodes  are  from 
three  to  five  logical  communications  links.  Each  logical 
link  contains  from  one  to  three  physical  transmission  lines 
(trunks),  again  depending  on  the  traffic  volume  of  the  site. 


If  all  trunks  of  each  PSN-PSN  link  were  terminated 
on  the  same  SCM,  the  routing  of  packets  within  the  node 
could  be  done  on  a  deterministic  basis.  However,  in  or¬ 
der  to  insure  that  the  load  is  evenly  distributed  over 
SCMs  at  the  node,  to  minimize  dual  processing  required 
because  of  node  connectivity,  and  to  provide  higher  access 
to  the  available  transmission  bandwidth,  the  trunks  for 
each  link  are  distributed  across  SCMs  at  each  node  (Ref  4i25). 
This  is  shown  in  Figure  2  with  the  bottom  figure  representing 
the  selected  method  of  tying  SCMs  together.  NOTEi  This 
figure  is  for  example  only.  Each  node  does  not  have  three 
SCMs  and  the  nodes  are  not  totally  connected  in  the  real 
network.  The  example  is  shown  to  convey  the  idea  of  even 
trunk  distribution. 

The  connectivity  of  the  Host  computers  and  terminals 
into  a  node  is  called  the  access  area.  The  term  backbone 
pertains  to  the  routing  of  packets  solely  between  nodes. 

There  are  four  basic  paths  for  packets  to  take  in  the  over¬ 
all  network  system i  (1)  access  area-to-access  area, 

(2)  access  area-to-backbone,  (3)  backbone -to-backbone,  and 
(4)  backbone-to-access  area.  These  paths  are  shown  in 
Figure  3»  In  Figure  3a »  packets  enter  the  node  from  the 
access  area,  processed,  and  routed  locally  back  to  the  access 
area.  Figure  3b  shows  packets  entering  the  node  from  the 
access  area,  being  processed  and  routed  to  another  node  via 
the  network  backbone.  Figure  3c  depicts  the  path  for  pac¬ 
kets  entering  the  node  from  another  node  via  the  network 
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Packet  Paths  at  a  Node 


backbone,  processed  and  routed  to  another  node  via  the 
backbone.  In  Figure  3d,  packets  enter  the  node  via  the 
backbone  from  another  node,  processed  and  routed  to  a 
local  destination  via  the  access  area. 
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Ill  An  Overview  of  Routing  Algorithms 


The  basic  philosophy  behind  a  message  or  packet 
routing  function  is  the  establishment  of  a  continous  path, 
usually  incorporating  several  transmission  lines  (trunks) 
of  a  network,  between  any  pair  of  source  and  destination 
nodes,  along  which  messages  are  transmitted  (Ref  5 i213)» 

The  implementation  of  the  route  chosen  consists  of  setting 
up,  at  each  node  along  the  path,  a  routing  table  that 
directs  messages  with  a  particular  destination  address  to 
the  appropriate  outgoing  trunk  at  the  node.  The  table  and 
associated  table  look-up  procedures  are  incorporated  in 
the  memory  and  software  of  the  computer  at  the  node  in 
question  (Ref  5«213), 

A  fundamental  element  underlying  all  routing  algorithms 
is  the  choice  of  the  control  ideology  under  which  the  algo¬ 
rithm  is  executed.  The  four  most  widely  known  and  accepted 
ideologies  are  non-adaptive,  centralized,  isolated,  and 
distributed  (Ref  6), 

Non-Adaptive  Control 

Non-adaptive  or  fixed  algorithms  are  deterministic 
(static)  in  nature,  that  is,  they  do  not  adapt  or  change  to 
take  into  consideration  vacillations  in  traffic  congestion 
or  topological  changes  at  the  node.  Rather,  they  are  de¬ 
signed  to  provide  satisfactory  performance,  on  the  average, 
over  a  range  of  traffic  intensities  (Ref  5*214).  A  message's 
path  through  the  network  is  uniquely  predetermined  from 

14 


Jl 


'pe  y  "■!  'y»  gr 


■ijuifc. 1 


its  source  node  to  its  destination  node.  Routing  tables, 
predefined  at  each  node,  contain  the  path  a  message  takes 
for  any  unique  source-destination  node  pair.  The  routing 
algorithm  generally  used  for  this  type  of  control  ideology 
is  called  a  shortest  path  type  of  algorithm.  For  each 
unique  source-destination  node  pair,  one  path  is  generated. 
The  path  is  broken  into  segments,  each  node  in  the  path 
being  responsible  for  routing  the  message  over  the  par¬ 
ticular  segment  it  is  in  charge  of.  SITA,  a  cooperative 
company  of  international  air  carriers,  has  established  a 
data  communications  network  that  employs  this  type  of 
routing  algorithm  (Ref  5i28). 

For  the  reader's  information,  AUTODIN  I  maintains 
static  (fixed)  routing  tables.  The  entries  to  the  routing 
tables  are  input  one  at  a  time,  by  hand-cut  paper  tape.  A 
routing  command/change  is  entered  at  the  operator's  console 
which  reads  the  routing  changes  on  the  paper  tape.  An  on¬ 
line  utility  program  makes  the  correct  changes  to  the  routing 
tables.  The  routing  table  consists  of  primary,  secondary, 
and  in  some  cases  tertiary  routes.  If  the  primary  route 
for  a  message  is  disabled,  an  ALTROUTE  (alternate  route) 
command  is  entered  from  the  operator's  console  which  auto¬ 
matically  establishes  the  secondary  route  as  the  primary 
route.  When  the  disabling  condition  is  corrected,  a  com¬ 
mand  is  entered  at  the  console  which  reestablishes  the 
primary  route. 
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Centralized  Adaptive  Control 

Centralized  adaptive  algorithms  utilize  some  kind  of 
central  supervisory  program  which  dictates  the  routing 
decisions  to  the  individual  nodes  in  response  to  network 
changes  (Ref  6).  TYMNET,  a  computer-communications  net¬ 
work  developed  in  1970,  executes  this  class  of  routing 
algorithm.  One  example,  the  least-cost  algorithm,  as  em¬ 
ployed  by  TYMNET,  is  executed  every  time  a  user  connects 
into  the  network  and  selects  the  route  determined  to  be 
the  least-cost  for  the  user,  then  transmits  this  selected 
route  to  the  nodes  involved  ahead  of  time  (Ref  5«27).  The 
routing  algorithm,  as  contained  in  the  supervisory  program, 
finds  the  path  of  current  least-cost,  summed  over  the  costs 
of  each  link  on  the  path,  from  source  computer  to  destination 
computer  (Ref  5  s 27 ) • 

Isolated  Adaptive  Control 

Isolated  adaptive  algorithms  operate  independently 
of  other  nodes  in  the  network  with  each  node  making  exclu¬ 
sive  use  of  local  data  to  adapt  to  changing  conditions 
(Ref  6).  The  data  usually  contain  parameters  on  source 
node,  destination  node,  local  trunk  queue  lengths,  local 
topology,  and  message  priority.  From  the  data,  a  delay 
table  and  a  routing  table  are  formed.  The  delay  time 
estimate  entries  in  the  delay  table  are  the  number  of 


( 


messages  waiting  in  the  queue  on  the  particular  outgoing 
link.  The  delay  table  is  indexed  by  the  destination  node 
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and  outgoing  links.  If  a  link  connecting  two  nodes  con¬ 
tain  more  than  one  trunk,  the  message  with  the  highest 
priority  is  always  routed  on  the  trunk  with  the  shortest 
queue.  For  non-priority  messages,  either  trunk  can  be  se¬ 
lected  if  the  difference  in  the  queue  lengths  is  not 
greater  than  a  set  weighted  bias.  The  bias  can  be  adjusted 
to  fit  the  needs  of  the  node  and  the  level  of  traffic  volume. 
From  the  delay  table,  the  link  with  the  smallest  delay  is 
inserted  in  the  routing  table.  The  routing  table  is  indexed 
by  the  destination  node  and  in  the  case  of  a  multi-trunk 
link,  the  trunks  with  the  shortest  and  next  shortest  queues. 

Distributed  Adaptive  Control 

By  far  the  most  widely  used  and  the  subject  of  current 
state-of-the-art  research,  is  the  distributed  adaptive  rout¬ 
ing  algorithm.  It  utilizes  internode  cooperation  and  ex¬ 
change  of  information  to  arrive  at  routing  decisions  (Ref  6). 
Several  techniques  are  combined  to  make  up  this  type  of 
algorithm.  First  of  all,  the  main  feature  of  the  algorithm 
involves  the  exchange  of  delay  information  between  neigh¬ 
boring  nodes.  Whether  the  delay  information  is  determined 
on  estimated  delay  times  to  all  the  network  nodes  or  just 
the  delay  to  neighboring  nodes,  it  is  this  information 
which  the  routing  algorithm  uses  to  update  local  routing 
tables  at  the  node.  Second,  added  to  these  delay  estimates 
are  local  delay  values  based  on  the  number  of  messages  or 
packets  on  queue  awaiting  transmission  on  local  trunks. 
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Third,  these  combined  delays  are  compared  against  the  cur¬ 
rent  corresponding  entries  in  the  routing  table  and  are 
substituted  for  the  current  entries  if  the  new  delays  are 
less  than  the  current  delay.  And  fourth,  the  process  of 
local  decision  routing  becomes  inherent  in  the  algorithm 
by  virtue  of  the  fact  that  local  queue  lengths  take  an 
active  part  in  the  delay  calculations.  The  ARPANET  employs 
a  version  of  this  type  of  routing  algorithm  with  delay  status 
messages  exchanged  with  neighboring  nodes  on  the  order  of 
every  625  milliseconds  (Ref  5*48). 

The  routing  algorithm  proposed  for  the  AUTODIN  II 
network  is  very  similar  to  the  ARPANET  algorithm  but  con¬ 
tains  four  major  differences.  First,  status  information 
is  exchanged  between  all  network  nodes  rather  than  with 
neighboring  nodes.  Second,  the  information  exchanged  re¬ 
flects  local  link  congestion  and/or  local  link  connectivity 
status,  that  is,  whether  or  not  the  link  is  up  or  down 
rather  than  the  estimated  delay  to  each  of  the  other  nodes. 

Third,  the  exchange  rate  of  status  packets  is  either  period¬ 
ical  (every  400  milliseconds),  event  driven  when  link  con¬ 
gestion  occurs,  or  when  a  change  in  link  status  has  been 
determined  rather  than  set  at  every  625  milliseconds.  And 
fourth,  changes  in  the  status  of  links  or  nodes  which  cause 
the  generation  of  status  packets  produce  a  network  topolo¬ 
gical  table  resident  at  each  node  making  that  node  cognizant 
of  the  status  of  each  of  the  other  nodes  in  the  network, 
whereas  the  ARPANET  does  not  maintain  a  topological  table  at  all. 
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IV  Hierarchical  Routing  Algorithm  Description 


Chapter  IV  describes  in  detail  the  different  tables 
generated  and  used  by  the  routing  algorithm.  It  also  con¬ 
tains  a  detailed  explanation  of  each  module  making  up  the 
algorithm.  After  reading  Chapter  IV,  the  reader  should 
have  a  good  grasp  of  how  the  algorithm  builds  and  uses  its 
various  tables  in  order  to  arrive  at  the  best  trunk  over 
which  to  route  a  packet  to  its  appropriate  destination. 

The  reader  should  also  have  a  feel  of  how  each  module  fits 
into  the  overall  operation  of  the  algorithm. 

The  hierarchical  routing  algorithm  is  a  distributed 
adaptive  algorithm  utilizing  both  status  information 
exchanged  between  nodes  in  the  network  and  local  output 
queue  lengths  to  determine  the  best  path  over  which  to 
route  a  packet.  It  derives  its  name  from  the  two  levels 
of  routing  procedures  executed  to  determine  the  final 
trunk  over  which  a  packet  is  transmitted. 

The  unique  quality  requiring  two  levels  of  decision 
making  is  due  to  the  presence  of  multiple  (one  to  three) 
communications  processors  called  Switch  Control  Modules 
(SCMs),  located  within  each  node  of  the  network.  At  present, 
AUTODIN  II  is  the  only  packet-switching  network  to  employ 
multiple  processors  to  handle  the  routing  of  packets 
through  a  network.  Conditioned  on  predicted  traffic 
volumes  per  site,  one  to  three  SCMs  (PDP  ll/?0s)  handle  the 
routing  decisions.  Neighboring  nodes  connected  by  one 
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logical  link  but  that  link  is  made  up  of  one  to  three  physi¬ 
cal  transmission  lines  (trunks). 

The  first  level  of  decision  making  selects  the  route 
or  logical  link  over  which  the  packet  will  be  transmitted. 
The  second  level  selects  which  SCM  to  use  which,  in  effect, 
selects  the  individual  trunk. 

When  a  packet  is  received  at  a  node,  some  process 
or  set  of  processes  must  be  executed  to  determine  the 
next  route  the  packet  has  to  take  to  reach  its  final  des¬ 
tination.  The  hierarchical  routing  algorithm  is  made  up 
of  such  a  set  of  processes  to  determine  the  best  route  for 
the  packet  to  follow.  The  algorithm  contains  processes  that 
maintain  and  update  various  tables  in  the  algorithm's 
database  used  to  calculate  the  best  route.  Route  conges¬ 
tion  or  failure  along  with  nodal  congestion  or  failure  are 
detected  by  the  algorithm  and  compensated  for  by  a  recal¬ 
culation  of  its  database  tables.  Network  awareness  of  the 
degradation  is  facilitated  by  the  cognizant  nodes  formu¬ 
lating  and  transmitting  status  messages  to  the  other  nodes 
in  the  network.  Upon  reception  of  the  status  messages,  a 
node  uses  the  information  in  the  status  messages  to  recal¬ 
culate  its  tables. 

Due  to  the  configuration  of  the  trunk  connections 
to  the  SCM(s),  different  routing  results  are  obtainable. 

In  one  instance,  a  node  contains  one  SCM  and  the  selected 
link  resulting  from  the  execution  of  the  algorithm  contains 
multiple  trunks.  The  trunk  that  has  the  shortest  queue  is 
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selected.  In  the  case  where  the  node  contains  multiple 
SCMs  and  the  selected  link  is  made  up  of  only  one  trunk, 
the  packet  could  incur  additional  delay  required  to  switch 
the  packet  to  the  SCM  that  is  connected  to  the  correct 
trunk.  That  additional  delay  is  not  incurred  if  the  SCM 
processing  the  packet  is  also  the  SCM  to  which  that  trunk  is 
connected.  In  the  situation  where  the  node  contains  two 
SCMs  and  the  selected  link  contains  two  trunks  one  connected 
to  each  of  the  SCMs,  a  bias  factor  is  involved  in  the  trunk 
selection  because  of  the  possibility  of  having  to  switch 
SCMs.  If  the  packet  is  routed  to  a  trunk  on  another  SCM 
within  the  same  node,  information  specifying  which  trunk 
queue  is  to  be  used  goes  along  with  it.  Thus  packets  ar¬ 
riving  at  an  SCM  from  another  SCM  within  the  same  node  can 
be  queued  directly  for  the  correct  trunks  (Ref  7«9)» 

The  operation  of  the  routing  algorithm  can  be  sum¬ 
marized  as  follows  (Ref  8il-2)i 

(1)  Each  node  maintains  a  copy  of  the  network 
topology. 

(2)  Each  node  monitors  its  output  trunk  group 
queue,  and  compares  them  with  a  predeter¬ 
mined  congestion  threshold  value. 

(3)  Whenever  the  immediate  connectivity  of  a  node 
alters,  or  the  congestion  status  of  any  of 
its  trunk  group  alters,  or  on  a  periodical 
basis,  the  node  transmits  a  connectivity/ 
congestion  message  to  all  other  nodes. 
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(4)  On  receipt  of  a  connectivity/congestion  mes¬ 
sage,  each  node  updates  its  local  copy  of  the 
network  topology  and  updates  its  table  of 
minimum-hop  paths  to  all  destinations. 

(5)  Tandem  nodes  (nodes  between  the  source  and 
destination  nodes)  route  packets  by  choosing 
only  among  the  minimum-hop  paths. 

(6)  Source  nodes  route  packets  by  choosing  among 
the  minimum-hop  paths  and  the  mimimum-hop  +  1 
paths . 

(7)  When  a  tandem  node  or  source  node  has  two  or 
more  paths  to  a  destination,  it  chooses  a 
specific  output  link  by  minimizing  the  com¬ 
bination  of  local  queue  delays  plus  bias  rep¬ 
resenting  the  delay  due  to  any  reported  con¬ 
gestion  on  the  paths. 

Algorithm  Database 

Each  node  operates  with  a  copy  of  the  algorithm.  There 
are  several  tables  and  arrays  calculated  and  maintained  at 
each  node  by  the  algorithm.  The  tables  are  the  Link  Distance 
(LD)  table  and  the  Routing  Distance  (D)  table.  The  LD  table 
represents  the  nodal  topology  of  the  network.  The  AUTODIN  II 
network  is  not  a  totally  connected  network,  that  is,  each 
node  is  not  connected  to  every  other  node.  The  LD  table 
maintains  which  nodes  are  connected  to  which  nodes  along 
with  the  identification  of  congested  links  between  nodes. 
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The  D  table  contains,  for  each  link  a  node  maintains,  the 
number  of  link  traversals  (hops)  necessary  to  reach  a 
given  destination  node. 

The  arrays  the  algorithm  maintains  and  uses  are  the 
Connectivity  Pointer  (P)  array,  Connectivity  Vector  (NC) 
array,  HOP  array,  LIST  array,  SROUTE  array,  and  the  ROUTE 
array.  The  P  and  NC  arrays  are  interrelated  arrays  used 
in  conjunction  with  other  arrays  to  build  the  D  table. 

The  HOP  array  contains  one  entry  per  node  in  the  network. 
Each  entry  is  the  number  of  hops  required  to  get  to  the 
respective  node  from  the  local  node.  The  LIST  array  also 
contains  one  entry  per  node  in  the  network.  Each  entry  is 
the  respective  node's  ID  number.  The  SROUTE  and  ROUTE 
arrays  are  used  to  select  the  link,  given  the  packet  des¬ 
tination  node,  over  which  to  route  the  packet.  SROUTE  is 
used  if  the  node  is  the  originating  node  of  the  packet  and 
ROUTE  is  used  if  the  node  is  an  intermediate  node. 

The  Link  Distance  (LD)  table  is  an  N  x  N  matrix 
where  N  is  the  number  of  nodes  in  the  network,  in  this 
case  eight.  Nodes  are  numbered  from  one  to  eight  in 
alphabetical  order  from  Albany  to  Tinker.  Source  nodes 
correspond  to  row  headings  and  destination  nodes  correspond 
to  column  headings.  An  entry  in  the  table  is  a  tri-state 
quantity  reflecting  either  connectivity  with  no  congestion, 
connectivity  with  congestion,  or  no  connectivity.  Por 
example,  let  node  1  and  node  j  be  two  nodes  in  the  network. 
If  there  is  a  link  between  them  and  it  is  not  congested, 
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the  LD  (i,j)  entry  is  1.0.  If  there  is  a  link  between  the 
two  nodes  but  it  is  congested  beyond  a  set  congestion 
threshold,  the  corresponding  LD  (i,j)  entry  is  a  1.01.  If 
the  two  nodes  are  not  connected  the  LD  (i,j)  value  is  a 
pseudo  infinity  value  (1000)  which,  in  practice,  is  a  very 
large  number  to  preclude  any  packet  from  being  queued  to  a 
non-existant  link.  Figure  4  is  representative  of  the 
initial  LD  for  the  Andrews  node. 

The  two  arrays  P  and  NC  are  interrelated  arrays. 

The  P  array  (connectivity  pointer  array),  is  an  N+l  x  1 
vector  whose  entries  point  to  entries  in  the  NC  array. 

The  NC  array  (connectivity  vector)  is  an  NL  x  1  vector 
where  NL  is  the  number  of  one  way  links  in  the  network, 
in  this  case  thirty- two.  The  first  NLN1  entries,  where 
NLN1  is  the  number  of  links  Albany  maintains,  are  the 
numerical  representations  of  the  nodes  connected  to  Albany. 
The  next  NLN2  entries,  where  NLN2  is  the  number  of  links 
Andrews  maintains,  are  the  numerical  representations  of 
the  nodes  connected  to  Andrews  and  so  on  for  the  rest  of 
the  nodes  and  their  respective  entries  in  the  NC  array. 

The  entries  of  the  P  array  point  to  the  entry  of  the  NC 
array  that  corresponds  to  the  first  link  of  a  particular 
node.  Figure  5  illustrates  the  contents  of  both  the  P  and 
the  NC  arrays. 

The  HOP  array  is  an  N  x  1  array  whose  entries  corres¬ 
pond  to  the  minimum  number  of  hops  required  for  a  packet  to 
reach  a  particular  node.  The  source  node*s  entry  is  a  zero. 
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Initially,  the  entries,  except  the  source  node  entry,  are 
either  Is  or  pseudo  infinities,  the  Is  corresponding  to 
directly  connected  nodes  and  the  pseudo  infinities  corres¬ 
ponding  to  non-connecting  nodes.  Figure  6  shows  both  the 
initial  HOP  array  and  the  revised  version  after  initial¬ 
ization  for  the  Andrews  node. 

The  LIST  array  is  an  NL  x  1  array.  Each  node  is 
given  a  unique  integer  identification  number  from  one  to 
eight.  The  ID  number  is  assigned  in  numerical  order  to  the 
nodes  arranged  in  alphabetical  order,  that  is,  Albany-1, 
Andrews-2,  ...,  Tinker-8.  The  first  NLN  entries,  where  NLN 
is  the  number  of  links  a  particular  node  maintains,  are  the 
numerical  representations  (ID  numbers)  of  the  nodes  connected 
to  the  particular  node  executing  the  algorithm.  They  are 
basically  taken  from  the  NC  array.  The  other  N  -  NLN  -  1 
entries  are  the  numerical  representations  of  the  nodes 
not  connected.  There  is  no  entry  for  the  source  node.  If, 
during  the  operation  of  the  algorithm,  a  connected  node's 
link  drops  out,  that  node's  numer'cal  value  is  negated, 
removed  from  the  set  of  connected  node's  value" 
serted  among  the  set  of  non-connected  nodes.  -  w- , 

that  link  is  flagged  as  being  disconnected.  Re. 

Figure  7  for  the  initial  LIST  array  and  the  revised  version 
after  initialization  for  the  Andrews  node. 

The  LD  table  and  the  P,  NC,  HOP,  and  LIST  arrays  are 
combined  to  form  the  most  important  table  the  algorithm 
maintains,  the  D  table  (routing  distance  table).  It  is  an 


27 


NoP(i) 


#0PU) 


Initial 


Revised 


IZST  (  i) 


izsru) 


Pig  7.  LIST  Array  for  Andrews 


29 


N  x  NLN  matrix,  where  again  N  is  the  number  of  nodes  in  the 
network  and  NLN  is  the  number  of  links  the  node  maintains. 

The  entries  reflect,  for  a  given  link,  the  number  of  hops 
(plus  the  congestion  factor  of  .01  if  congestion  exists) 
required  to  reach  a  particular  node.  The  table  contains 
only  minimum  hop  or  minimum  hop  +  1  values.  Minimum  hop  is 
defined  as  the  least  number  of  hops  necessary  for  a  packet 
to  make,  given  the  present  node  and  the  destination  node. 
Minimum  hop  +  1  is  just  one  more  than  minimum  hop.  Because 
a  node  is  made  up  of  multiple  links,  the  number  of  hops  a 
packet  would  be  required  to  make  on  one  link  could  be  more 
or  less  than  the  packet  would  be  required  to  make  if  routed 
on  another  link.  If  the  number  of  hops  required  for  a  given 
link  is  more  than  the  minimum  hop  +  1  count,  that  link’s 
entry  in  the  D  table  is  the  pseudo  infinity  value  to  pre¬ 
clude  its  usage  in  the  routing  calculations.  Thus,  each  row 
of  the  D  table  gives  the  minimum  hop/minimum  congestion 
"distance"  to  a  particular  destination  as  a  function  of  an 
outgoing  link,  such  that  only  minimum  hop  and  minimum  hop  +  1 
paths  have  non-infinite  values.  Figure  8  shows  the  initial 
D  table  and  the  revised  D  table  after  initialization  as  cal¬ 
culated  for  the  Andrews  node.  Note  the  initial  values  are 
similar  to  the  constructs  of  the  HOP  array,  that  is.  Is 
and  pseudo  infinities*  The  source  node  entries  are  set  to 
Os  for  the  set  of  outgoing  links. 

The  SROUTE  and  ROUTE  arrays  are  N  x  1  arrays.  An  entry 
in  either  array  represents  the  corresponding  link  to  select. 
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given  the  destination  node  of  the  packet.  If  the  node 
processing  the  packet  is  the  source  node,  the  SROUTE 
array  is  used  for  link  selection.  If  the  node  is  a 
tandem  ’node  (a  node  other  than  the  source  node  or  the  des¬ 
tination  node),  the  ROUTE  array  is  used.  The  rationale 
behind  having  two  link  routing  arrays  is  based  on  the  idea 
that  under  normal  operating  conditions,  a  packet  will  en¬ 
counter  at  the  most  only  one  tandem  node  because  of  the 
network  connectivity.  The  source  node  could  have  multiple 
"best"  routes  to  select  from  to  get  the  packet  to  its  des¬ 
tination,  but  a  tandem  node,  in  general,  would  have  only 
one  "best"  route  which  would  be  the  link  directly  connect¬ 
ing  it  to  the  destination  node. 


Modules  of  the  Algorithm 

The  hierarchical  routing  algorithm  is  made  up  of 
six  modules.  They  are  PAKROUTE,  MESGEN,  UPDATE,  RTABLE , 
TTABLE,  and  MINHOP.  The  overall  organization  of  the 
hierarchical  routing  algorithm  is  shown  in  Figure  9.  The 
box  representing  the  UPDATE  module  includes  the  RTABLE  and 
TTABLE  modules.  The  purpose  of  the  AUTHENTICATION  blocks 
is  to  insure  that  only  actual  nodes  (not  users)  can  send  an 
authentic  network  status  message.  Otherwise  a  malfunc¬ 
tioning  or  malicious  user  could  bring  down  the  network  by 
declaring  all  links  down  (Ref  8iA-3)<  The  "Background 
Functions”  and  "Per  Packet  Functions”  are  discussed  later  in 
this  chapter.  Table  I  should  be  consulted  for  definition  of 


TABLE  I 

Table  of  Variables 

Nominal  Value 


NN 

Number  of  nodes  in  the 
network 

8 

NL 

Number  of  one-way  links 
in  the  network 

32 

NLN 

Number  of  links  outgoing 
from  a  particular  node 

3-5 

INF 

Pseudo  infinity  number  > 
maximum  number  of  hops 
in  network 

1000 

LD(i.j) 

Link  distance  matrix 

NN  X  NN 

INF, 1,1. 01 

SELF 

Node  self  ID  number 

1-8 

D(i, j) 

Routing  Distance  table 

NN  X  NLN 

- 

P(i) 

Connectivity  Pointer  array, 

( NN+1 )  X  1 

- 

NC  (i) 

Connectivity  vector,  NL  X  1 

ROUTE(i) 

Tandem  node  routing  array 
(outgoing  link  ID  number  in¬ 
dexed  by  destination  node) 

HOP(i) 

Hop  count  array,  NN  X  1 

- 

LIST(i) 

Connectivity  set  array,  NL  X 

1 

BNOM 

Congestion  weight 

7 

CNOM 

Extra  hop  weight 

7 

STAT(i) 

Node  report  status  array, 

NN  X  1 

INF, 1,1. 01 

T1 

Congestion  threshold  (pkts 
per  trunk) 

7 

T2 

Saturation  threshold  (1,2, or 
SCMs)  total  number  of  packets 
in  input  queues 

3  7,18,20 

TBUFF 

Total  number  of  packets  in 
input  queues 

• 

BUFF( i ) 

Number  of  packets  queued  on 
link  i 

— 

LINES 

Number  of  56Kbps  trunks  serving  1-3 
link  i 

TABTICK 

Time  interval  for  running  RTABLE  100  msec 

34 


TABLE  I  (CONT'D) 


PTICK 

STICK 

SROUTE(i) 

ZONE 

BUP(i) 


Time  interval  for  "BAD" 
news  detection  in  MESGEN 

Time  interval  for  "GOOD" 
news  detection  in  MESGEN 

Source  node  routing  array 
(outgoing  link  ID  number 
indexed  by  destination  node), 
NN  X  1 

Flag  denoting  the  status  of 
the  node  on  previous  entry 
into  MESGEN  (0  -  not  saturated 
1  -  saturated) 

Previous  status  of  link  array, 
NLN  X  1  (1  -  link  up,  1000  - 
link  down) 


Nominal  Value 
100  msec* 

400  msec* 

0,1 

1,1000 


variables  used  in  the  following  figures  as  the  algorithm 
is  explained.  The  symbol  Z  in  the  figures  is  used  to  de¬ 
note  a  sleep  or  idle  point  for  a  procedure. 

i 

PAKROUTS.  The  module  executed  for  every  packet  pro¬ 
cessed  is  PAKROUTE,  Figure  10.  Depending  on  the  destina¬ 
tion  of  a  packet,  a  link  is  selected  from  the  SROUTE  or 
ROUTE  array  (processing  node  is  the  source  or  tandem  node 
of  the  packet).  When  the  link  contains  only  one  trunk,  the 
packet  is  queued  for  output  on  that  trunk.  If  the  selected 
link  contains  more  than  one  trunk,  the  trunk  with  the 
shortest  output  queue  is  selected. 

MESGEN.  MESGEN  is  described  in  Figure  11.  The  prime 
function  of  MESGEN  is  the  generation,  when  necessary,  of 
status  packets  containing  the  node's  current  connectivity 
or  congestion  changes.  A  link  is  declared  congested  when¬ 
ever  the  queued  packets/ trunk  for  a  link  exceeds  a  threshold 
Tl.  A  node  is  considered  to  be  saturated  whenever  the  total 
of  all  input  queued  packets  exceeds  a  threshold  T2.  It  is 
entered  periodically  at  two  different  time  intervals,  FTICK 
(100  milliseconds)  and  STICK  (400  milliseconds).  When  en¬ 
tered  as  a  result  of  FTICK,  "bad  news"  is  checked.  Links 
are  checked  to  see  if  any  have  gone  down  since  the  last 
FTICK  entry.  Congestion  on  the  individual  links  is  also 
checked.  If  none  of  the  links  have  gone  down,  or  there  is 
no  congestion  on  any  of  the  links,  then  node  saturation  is 
checked.  If  the  saturation  threshold  is  equalled  or  sur- 
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passed  and  the  node  was  not  saturated  at  the  last  FTICK 
entry  (ZONE  =  0),  the  node  is  considered  saturated  (Zone  = 

1)#  and  a  status  message  declaring  all  the  node's  links 
congested  is  formulated.  If  one  or  more  links  are  down  or 
congestion  exists  on  one  or  more  links  a  status  packet  is 
formulated  selectively  indicating  congestion  on  the  link  or 
links  in  question.  A  copy  of  the  status  packet  is  sent  to 
UPDATE  for  local  processing.  Copies  of  the  status  packet 
are  also  sent  to  other  network  nodes  for  their  individual 
processing. 

Upon  entry  because  of  the  STICK  time  interval,  "good 
news"  is  checked.  Links  are  checked  to  see  if  any  have 
come  up  as  well  as  the  absence  of  congestion  on  any  of  the 
links  since  the  last  STICK  entry.  If  the  status  of  the  links 
are  the  same  as  the  previous  entry  time  interval,  the  satur¬ 
ation  condition  of  the  node  is  checked.  If  the  node  is  cur- 
rently^not  saturated  and  was  saturated  upon  the  last  STICK 
entry  into  MESGEN  (ZONE  =  1),  the  node  is  considered  not 
saturated  (ZONE  *  0).  Even  though  the  node  is  considered 
unsaturated ,  some  links  still  might  be  congested,  so  they 
are  checked  individually  for  congestion  in  the  event  a  link 
congestion  status  message  is  needed.  If  a  link  has  come  up 
or  congestion  no  longer  exists  on  a  link,  a  status  packet  is 
generated  by  selectively  checking  each  link.  A  copy  of  the 
status  packet  is  sent  to  UPDATE  for  local  processing.  Ad¬ 
ditional  copies  are  sent  to  other  nodes  for  their  processing. 

Under  entry  into  MESGEN  because  of  FTICK  or  STICK,  if 
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the  processing  gets  down  to  check  for  the  presence  or  ab¬ 
sence  of  node  saturation  and  the  result  of  the  check  is 
negative,  the  module  is  exited  with  no  further  processing. 
Figure  11  is  the  corrected  copy  of  the  original  module.  An 
error  was  found  during  the  code  conversion  to  Fortran. 

The  blocks  explaining  the  actual  entries  into  the  status 
packet  had  been  interchanged. 

UPDATE.  UPDATE  is  executed  on  each  arrival  of  a  status 
packet.  As  implemented  in  this  simulation,  for  each  pair 
of  nodes,  the  LD  table  contains  an  entry  indicating  either 
the  nodes  are  connected  (1.00)  and  congested  (1.01),  or  not 
connected  (1000).  Each  arriving  status  message  contains  its 
source  node’s  row  of  the  LD  table.  The  LD  table  is  the  tab¬ 
le  that  is  initially  updated  when  a  node  receives  a  status 
message.  The  source  node  of  the  status  packet  encodes  its 
corresponding  row  entries  of  the  LD  table  and  broadcasts  it 
to  the  other  nodes.  Upon  reception  of  the  status  message, 
the  respective  nodes  compare  the  values  in  the  status  mes¬ 
sage  against  the  appropriate  row  entry  in  their  own  LD  table 
If  the  values  are  the  same  the  status  message  is  discarded. 
If  the  values  are  different,  the  information  in  the  status 
message  replaces  the  corresponding  row  of  the  LD  table.  Fig 
ure  12  describes  the  procedure  of  UPDATE.  It  should  be 
noted  that  the  actual  algorithm  employs  a  2-bit  per  entry 
scheme  in  the  status  message.  In  binary  form  00  represents 
not  connected,  01  represents  connected  but  no  congestion, 
and  11  represents  connected  and  congestion  exists.  In  the 
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simulation,  each  entry  in  the  status  message  uses  one 
word  to  represent  the  above  with  entries  of  1000,  1.00, 
and  1.01  respectively. 

RTABLE.  RTABLE  is  shown  in  Figure  13*  The  function 
of  RTABLE  is  to  update  the  source  and  tandem  node  routing 
arrays,  SR0UTE  and  ROUTE.  Periodically,  and  after  each 
execution  of  MINH0P,  RTABLE  is  entered  to  recompute  the 
routing  arrays  using  the  more  current  D  table  and  local 
queue  lengths. 

TTABLE .  TTABLE  is  the  module  used  to  build  the  trunk 
routing  tables.  The  tables  contain  the  designated  trunks 
over  which  packets  are  to  be  routed,  given  a  selected  out¬ 
put  link  from  SROUTE  or  ROUTE.  The  implementation  of  the 
routing  algorithm  in  the  simulation  model  did  not  execute 
TTABLE.  Instead,  the  length  of  the  queues  of  each  trunk 
in  the  selected  output  link  is  scanned  to  select  the  trunk 
with  the  shortest  queue.  That  same  procedure  is  used  in  the 
execution  of  TTABLE  in  any  event,  only  the  extra  code  to 
actually  build  and  maintain  the  trunk  routing  tables  TTABLE 
was  not  used. 

MINHOP.  MINH0P  calculates  the  distance  table  (D  tab¬ 
le)  indexed  by  outgoing  links  and  final  destination  node. 
Table  entries  encode  (conditioned  on  outgoing  links  and 
destination  node)  the  minimum  hop  count  and  minimum  con¬ 
gestion  along  minimum  hop  paths.  Entries  are  pseudo  infinite 
values  if  the  hop  count  (for  outgoing  links  and  destination 
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node)  exceeds  the  overall  (for  destination  node)  minimum  hop 
count  plus  one  (Ref  7i7). 

MINHOP  is  the  heart  of  the  routing  algorithm.  It  can 
be  described  in  four  procedures i  (1)  CONNECTIVITY,  (2)  INI- 
TIALIZE_D,  (3)  COMPUTER),  and  (4)  CONN__CHANGE .  CONNEC¬ 
TIVITY,  Figure  14,  converts  the  connectivity  information  in 
the  link  distance  (LD)  table  into  more  convenient  forms  of 
the  P  and  NC  arrays •'  It  is  entered  only  once,  during  ini¬ 
tialization.  INITIALIZE^ D,  Figure  15,  initializes  the  D 
table  from  the  values  of  the  LD  table,  P  and  NC  arrays. 
COMPUTE_D,  Figure  16,  executes  a  search  process  to  fill  out 
the  rest  of  the  D  table.  The  above  two  procedures  are  en¬ 
tered  if,  after  a  status  packet  is  received,  an  update  to 
the  D  table  is  required.  In  the  Interest  of  clarification, 
any  further  references  to  MINHOP  will  be  synonymous  with  the 
execution  of  INITIALIZE_D  and  COMPUTEJ).  CONN_ CHANGE ,  Fig¬ 
ure  17,  updates  the  NC  array  to  reflect  link  failure  or 
repair.  In  addition,  it  updates  the  LD  table  which  stores 
the  link  distance  and  congestion  information  (Ref  7»27)» 
CONN_CHANGE  is  entered  when  the  status  of  a  link  changes. 

Background  Functions 

Figure  18  pictorially  describes  the  functions  per¬ 
formed  by  the  algorithm  in  the  Background  mode.  The  ac¬ 
tuation  of  a  particular  procedure  is  either  periodical  or 
event  driven.  In  the  Background  mode  the  tasks  MINHOP, 
RTABLE,  TTABLE,  UPDATE,  and  MESGEN  are  executed  periodically. 
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If  it  is  time  to  execute  one  of  the  above  modules,  the  pro¬ 
cessor  enters  a  multi-processing  mode  where  it  can  process 
a  packet  and  execute  the  appropriate  module  at  the  same  time 

The  King  determination  process  was  not  incorporated 
into  the  algorithm  as  simulated,  but  will  be  incorporated 
into  the  ON-LINE  version  of  the  algorithm.  The  background 
tasks  of  MINHOP,  RTABLE ,  TTABLE ,  UPDATE  and  MESGEN  need 
to  be  done  only  at  one  SCM  per  node.  One  SCM  will  thus  be 
designated  as  the  "King",  and  have  responsibility  for  these 
tasks  (Ref  ?i4).  The  King  will  be  periodically  determined 
in  a  distributed  fashion  as  the  SCM  having  the  minimal  load. 
In  the  case  of  a  failure  of  a  King  processor  the  next  King 
determination  will  reestablish  order  (Ref  ?i4).  King  deter¬ 
mination  will  be  interrupt  initiated  and  will  be  reassigned 
only  upon  failure  of  the  current  King  processor. 

To  supply  the  King  with  the  necessary  queue  information 
the  SCMs  will  periodically  exchange  information  vectors  con¬ 
taining  (Ref  ?i4)i 

(1)  The  processor  utilization  during  a  specified  time 
interval  or  other  estimates  of  the  current  pro¬ 
cessor  load. 

(2)  The  input  queue  size  for  processing  at  the  SCM. 

(3)  The  output  queue  sizes  for  each  of  the  trunks 
emanating  from  the  SCM  having  the  minimum  pro¬ 
cessor  utilization. 

Once  the  King  has  been  established,  the  queue  sizes 
for  trunks  are  consolidated  as  queue  sizes  for  links  and 


the  King  assumes  command  of  the  execution  of  MINHOP, 


RTABLE,  UPDATE  and  MESGEN.  The  "Coronation  Process" 
must  be  executed  identically  at  multiple-SCM  nodes  in 
order  for  each  SCM  to  arrive  at  the  same  SCM  to  choose  as 
King.  At  any  time,  the  King  might  die  (at  the  very  least 
hardware  errors  are  very  unpredictable),  therefore,  the 
other  SCMs  at  a  node  must  maintain  the  information  necessary 
to  assume  command. 

Per  Packet  Functions 

Figure  19  shows  a  schematic  view  of  the  per  packet 
routing  tasks.  The  routing  tables  produced  by  RTABLE  are 
used  to  determine  which  outgoing  link,  if  any,  is  to  be 
used.  PAKROUTE  splits  the  packets  into  three  broad  cate¬ 
gories!  (1)  Packets  destined  for  other  nodes,  (2)  pac¬ 
kets  destined  for  the  present  node,  and  (3)  undeliverable 
packets  (due  to  hardware  failures  separating  part  of  the 
network,  etc.)  (Ref  4i40).  The  per  packet  function  mode 
was  not  entirely  incorporated  into  the  algorithm  as  exer¬ 
cized  in  the  simulation.  The  use  of  the  Processor  Communi¬ 
cations  Link  (PCL)  was  not  implemented.  When  a  packet 
enters  a  node,  PAKROUTE  is  executed  by  the  individual  pro¬ 
cessor  processing  the  packet.  As  a  result  of  PAKROUTE,  the 
packet  is  then  queued  directly  to  whichever  trunk  was  se¬ 
lected.  The  functions  as  portrayed  in  Figure  19  will  be 
iaplemented  into  the  ON-LINE  version  of  the  algorithm. 

As  explained  in  Chapter  VII,  the  hierarchical  routing 
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algorithm  was  written  in  the  high  order  programming  lan¬ 
guage  FORTRAN  IV.  The  rest  of  the  simulation  model  was 
coded  in  the  simulation  language,  General  Purpose  Simulation 
System  (GPSS).  The  following  chapter  provides  a  description 
of  GPSS  along  with  a  detailed  description  of  the  simulation 
code  for  one  of  the  nodes  in  the  network. 
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V  GPSS  SIMULATION  MODEL 


Language  Description 

GPSS  (General  Purpose  Simulation  System)  was  developed 
in  1961  by  Geoffrey  Gordon.  The  primary  purpose  in  devel¬ 
oping  such  a  simulation  language  was  to  develop  a  simulation 
tool  for  use  by  systems  analysts  who  were  not  adept  in  com¬ 
puter  programming.  Because  of  the  universality  of  flow 
charts,  GPSS  was  structured  as  a  block-oriented  language. 

This  idea  allows  the  modeler  to  construct  the  model  of  a 
system  using  blocks  which  are  connected  together  to  form  a 
network.  The  blocks  are  connected  together  in  the  same  se¬ 
quence  of  events  as  occur  in  the  system  being  modeled  (Ref  9). 
A  set  of  unique  block  types  are  defined,  each  corresponding 
to  a  basic  system  action.  In  this  way,  system  actions  and 
the  times  at  which  those  actions  occur  are  modeled  by  drawing 
a  block  diagram  of  the  system  and  inserting  the  appropriate 
block  instruction  to  accomplish  the  desired  action.  The 
components  of  the  model  on  which  the  system  acts  are  called 
transactions.  They  in  turn,  model  the  dynamic  structures 
of  the  system.  Transactions  move  through  the  block  struc¬ 
tured  program,  being  created,  modified,  delayed,  transferred, 
or  destroyed  as  required. 

Although  GPSS  was  developed  to  model  and  simulate  a 
vast  scope  of  various  real  world  physical  and  abstract 
systems,  it  particularly  lends  itself  to  the  discrete-event 
simulation  of  queueing  systems.  The  natural  order  of  en- 


tities,  instructions,  model  description  and  internal 
organization  of  the  simulator  allows  a  queueing  model  to 
be  developed,  coded,  and  simulated  almost  effortlessly. 

Transactions,  as  described  earlier,  sire  the  principle 
source  of  traffic  flowing  through  the  model.  They  represent 
some  operation  of  the  system  being  simulated  but  the  modeler 
defines  what  those  operations  are.  Transactions  move  from 
block  to  block  in  a  manner  similar  to  units  of  traffic  in 
the  real  system  they  represent.  Each  movement  is  considered 
to  be  an  event  that  is  due  to  occur  at  some  point  in  time. 

GPSS  automatically  maintains  a  sequential  record  of  the 
events  which  are  to  occur.  In  those  cases  when  events  (such 
as  attempting  to  seize  a  facility  which  is  already  in  use) 
cannot  take  place  at  the  instant  in  time  when  they  were 
originally  scheduled,  the  transaction  ceases  to  move  until 
the  event  can  take  place.  During  a  transaction's  scheduled 
movement,  it  moves  through  as  many  blocks  as  it  can  before 
being  denied  entry  to  a  block.  The  program  maintains  the 
status  of  the  condition  causing  the  transaction  to  be  delayed. 
As  soon  as  the  condition  changes,  the  transaction  proceeds 
in  the  appropriate  sequence  with  respect  to  the  other  trans¬ 
actions,  that  is,  the  transactions  scheduled  to  move  first 
do  so.  Associated  with  each  transaction  are  attributes 
called  parameters.  These  parameters  enable  the  modeler  to 
keep  track  of,  collect,  and  modify  dynamically  those  charac¬ 
teristics  which  are  of  particular  interest  to  the  modeler 
about  that  particular  transaction. 
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The  GPSS  model  is  made  up  of  blocks  correlating  to  the 
block  diagram  constructed  to  define  the  modeled  system. 

Each  block  can  be  thought  of  as  a  small  subroutine  or 
macro  instruction.  Each  block  is  associated  with  infor¬ 
mation  falling  into  three  categories t  location,  operation, 
and  operands  (Ref  10).  Every  block  occupies  a  specific 
location  in  the  block  diagram.  The  first  block  in  a  model 
occupies  location  1,  the  second  block  occupies  location  2, 
and  so  on.  The  GPSS  Processor  automatically  assigns  a 
location  number  to  each  block  unless  the  user  option  is 
utilized  in  which  the  modeler  labels  a  particular  block 
with  a  name  of  his  own  choice  containing  from  three  to 
five  alphanumeric  characters,  the  first  three  of  which 
must  be  alpha.  The  block's  number,  or  label,  can  be  refer¬ 
enced  in  later  blocks.  The  "operation"  of  a  block  is  a 
"verb"  suggestive  of  the  task  the  block  accomplishes. 

Each  block  type  is  characterized  by  its  own  predefined 
verb,  that  is,  GENERATE,  ASSIGN,  QUEUE,  SEIZE,  and  TRANS¬ 
FER  just  to  name  a  few.  A  block  has  operands  whose  values 
identify  the  specific  actions  the  block  is  to  take.  The 
operands  can  be  conveniently  thought  of  as  arguments  used 
in  subroutine  calls.  Four  basic  types  of  events  occur 
with  a  blocki  (1)  creation  or  destruction  of  a  transaction, 

(2)  alteration  of  a  numerical  attribute  of  a  transaction, 

(3)  delay  of  a  transaction  for  some  specified  amount  of 
time,  and  (4)  alteration  of  the  transaction  flow  through 
the  model. 
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i  There  are  four  categories  of  block  types.  They  are 

transaction-oriented,  transaction  flow-modifying,  equip¬ 
ment-oriented,  and  statistical.  Transaction-oriented 
blocks  deal  with  the  transaction  itself.  A  transaction  can 
be  created  as  in  the  GENERATE  block,  destroyed  as  in  the 
TERMINATE  block,  delayed  as  in  the  ADVANCE  block,  have  its 
parameters  modified  as  in  the  ASSIGN  block,  or  have  its 
priority  changed  as  in  the  PRIORITY  block.  Flow-modifying 
blocks  modify  the  flow  of  transactions  through  the  model. 

The  TRANSFER  block  is  generally  used  to  direct  a  trans¬ 
action  to  a  nonsequential  next  block.  The  LOOP  block 
causes  the  transaction  to  loop  through  a  specified  set  of 
blocks  until  a  condition  is  met  or  satisfied.  The  TEST 
i  block  allows  transfer  of  a  transaction  upon  testing  a 

specified  condition  which  is  the  result  of  an  algebraic 
comparison  between  two  operands  of  the  block.  Equipment- 
oriented  blocks  define  how  a  transaction  controls  the 
simulated  device,  facility,  equipment,  process  or  other 
entity  the  transaction  is  currently  using.  The  SEIZE 
block  indicates  the  use  of  a  facility  by  the  entering 
transaction  so  that  the  facility  remains  in  use  until  the 
seizing  transaction  releases  the  facility  by  entering  a 
RELEASE  block.  When  a  transaction  with  high  priority 
enters  a  PREEMPT  block,  it  suspends  the  progress  of  the 
lower  priority  transaction  which  had  previously  seized 
^  the  facility.  The  higher  priority  transaction  gains  con¬ 

trol  of  the  facility  and  when  it  is  through  with  the  facility. 
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returns  control  back  to  the  preempted  transaction  by  en¬ 
tering  a  RETURN  block.  If  some  type  of  storage  entity 
is  being  modeled,  a  transaction  entering  an  ENTER  block 
obtains  a  specified  number  of  units  of  the  storage  entity. 
When  the  used  storage  is  to  be  returned,  the  transaction 
enters  a  LEAVE  block.  Statistical  block  types  cause 
specified  statistics  to  be  gathered  and  stored  pertaining 
to  the  desired  program  entity.  When  a  transaction  enters 
a  QUEUE  block,  queue  statistics  on  that  transaction  such  as 
the  number  of  transactions  currently  in  the  queue,  maximum 
number  of  transactions  in  the  queue  at  any  one  time,  average 
time  for  a  transaction  to  spend  in  the  queue,  and  the  number 
of  transactions  that  incurred  no  wait  time  spent  in  the 
queue  are  activated.  The  DEPART  block  removes  the  trans¬ 
action  from  the  queue  and  causes  the  actual  statistics 
discussed  above  to  be  gathered  and  stored  for  later  output 
at  the  end  of  the  simulation.  The  blocks  discussed  above 
are  by  no  means  all  the  blocks  available  but  do  lay  the 
fundamental  groundwork  needed  to  understand  the  simulation 
model  to  be  discussed  shortly. 

Output  from  the  simulation  includes  statistics  on 
the  queues,  facilities,  storages,  specifically  defined 
program  entities  called  SAVEVALUEs,  and  any  tabular  data 
generated  by  transactions  entering  a  TABULATE  block. 

The  following  section  describes  the  basic  program 
used  to  simulate  the  AUTODIN  II  simulation  model. 


61 


Program  Description 


Two  versions  of  the  simulation  program  were  used.  Chap¬ 
ter  VI  explains  the  specific  details  for  the  Baseline  or  con¬ 
trol  program  using  fixed  routing.  Chapter  VII  explains  the 
incorporation  of  the  hierarchical  routing  algorithm  into  the 
simulation  model. 

The  simulation  program  models  the  eight  node  AUTODIN  II 
communication  computer  network.  The  eight  nodes  are  Albany, 
Andrews,  Ft.  De trick,  Gentile,  Hancock,  McClellan,  Norton, 
and  Tinker.  Figure  20  shows  how  the  nodes  are  connected 
together,  the  number  of  SCMs  at  each  node,  and  how  the  trunks 
are  connected  to  the  SCMs  at  the  nodes.  In  general,  the 
processing  and  routing  of  packets  at  each  node  is  the  same. 
The  simulation  program  must  generate  packets  (simulating 
packets  received  from  locally  connected  Host  computers  and 
terminals),  assign  the  appropriate  priority,  length,  source 
and  destination  node  identification  number  to  each  packet, 
and  queue  and  transmit  the  packet  to  the  appropriate  node 
(which  will  either  be  the  destination  node  or  an  intermediate 
tandem  node).  Figure  21  is  typical  of  the  program  block 
diagram  for  each  node.  The  number  of  SCM  processors,  queues, 
and  trunks  a  node  has  is  peculiar  to  each  node.  In  this 
example,  the  block  diagram  is  shown  for  the  Andrews  node  in 
the  Baseline  simulation  model.  The  packets  are  generated 
with  some  mean  interarrival  rate  peculiar  to  each  node  and 
are  exponentially  distributed.  In  Figure  21,  the  GENERATE 
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block  uses  the  FUNCTION  EXPON  to  determine  the  exponential 
distribution  of  the  packets*  Random  number  generator  1, 

RN1,  is  used  to  generate  a  random  number  from  .00000  to 
1.00000.  This  random  number  is  then  used  to  select  which 
multiplication  factor  (from  the  FUNCTION  EXPON)  to  multiply 
against  the  given  mean  interarrival  rate  (1st  operand  of 
GENERATE  block)  of  the  packet.  The  result  of  the  multipli¬ 
cation  becomes  the  number  of  time  units  (in  this  case  milli¬ 
seconds)  which  must  pass  before  the  next  packet  is  generated. 
The  packet  priority,  packet  length,  and  source  node  identi¬ 
fication  number  are  assigned  to  the  packet's  second,  third, 
and  fourth  parameter  values  respectively  by  the  ensuing 
ASSIGN  blocks.  The  packet's  destination  node  identification 
(ID)  number  is  generated  using  FUNCTION  DEST2 •  Random  num¬ 
ber  generator  two,  RN2,  is  used  to  generate  a  random  num¬ 
ber  from  .00000  to  1.00000.  That  number  is  used  to  select 
a  number  from  1  to  8  which  becomes  the  destination  node  ID 
number.  It  is  stored  in  parameter  1  of  the  packet. 

The  packet  is  transferred  to  either  SCM  1  (ANDY1)  or 
SCM  2  (ANDY2)  where  it  is  queued  for  input  processing.  For 
ease  of  explanation,  SCM  1  is  assumed  to  process  the  packet. 
When  the  processor  is  available,  the  queued  packet  seizes 
it  and  the  packet  departs  the  queue.  FUNCTION  PRI21  uses 
the  value  in  parameter  2  of  the  packet  (1  for  non-priority 
and  2  for  priority)  to  determine  how  much  processing  time 
the  packet  acquires  (5  milliseconds  for  non-priority  and  3 
milliseconds  for  priority  packets).  The  packet  releases 
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the  processor  and  moves  into  the  SAVEVALUE  blocks.  In  this 
simulation  program,  the  SAVEVALUE  blocks  are  used  to  store 
the  lengths  of  the  output  trunk  queues.  The  SAVEVALUEs  are 
used  later  on  in  the  program  to  assist  in  determining  which 
trunk  to  route  the  packets  over.  SAVEVALUEs  8  through  13 
are  used  for  saving  the  queue  lengths  at  the  Andrews  node. 
Everytime  a  packet  is  input  processed  (accomplished  by  the 
packet  executing  this  sequence  of  blocks),  new  current 
values  of  the  output  queue  lengths  are  stored  in  the  SAVE¬ 
VALUEs.  When  the  packet  enters  the  following  TRANSFER 
block,  packet  parameter  1  (destination  node's  ID  number)  is 
used  in  FUNCTION  R0UT2  to  select  the  link  over  which  to 
route  the  packet.  In  the  case  parameter  l's  value  is  a  2, 
the  packet  is  delivered  locally.  When  a  packet  terminates 
locally,  the  delay  time  the  packet  has  acquired  from  time 
of  generation  to  its  delivery  at  its  destination  is  tabulated 
(by  the  TABULATE  block)  in  the  system  delay  table  RTIME. 
Depending  on  its  priority  and  length,  the  packet's  delay 
time  is  further  tabulated  in  either  the  RTHIS  table  (priority 
-  short),  RTHIL  table  (priority  -  long),  RTLOS  table  (non¬ 
priority  -  short)  or  the  RTLOL  table  (non-priority  -  long). 
The  packet  then  enters  a  TERMINATE  block  where  it  is  removed 
from  the  system.  For  the  example  that  a  packet  is  to  be 
routed  to  another  node,  suppose  the  packet's  destination 
is  Tinker  (ID  =  8).  The  packet  enters  the  TEST  block  labeled 
ANQL4.  Since  the  Tinker  link  contains  two  trunks,  the  de¬ 
cision  has  to  be  made  which  trunk  to  select.  The  TEST  block 
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compares  the  two  queue  lengths  (ANQ41  and  ANQ42)  and  the 
packet  enters  either  the  queue  ANQ41  or  the  queue  ANQ42. 

Again  for  ease  of  explanation,  assume  the  packet  enters  the 
queue  ANQ42.  The  selected  trunk,  ANTK5 »  is  seized  when 
available  and  the  packet  departs  ANQ42.  Based  on  the  packet 
length  value  in  parameter  3  (1  -  short,  2  -  long),  and  the 
FUNCTION  ANSE4,  the  packet  acquires  either  27  milliseconds 
if  a  short  packet  or  100  milliseconds  if  a  long  packet  when 
it  enters  the  appropriate  ADVANCE  block.  Having  acquired 
the  correct  transmission  time,  the  packet  releases  the  trunk 
and  transfers  to  Tinker's  SCM  3  where  it  is  further  processed 
and  delivered. 

The  following  chapter  explains  the  specifics  on  how  the 
mean  interarrival  rates  are  determined,  where  the  values  for 
processing  and  transmission  come  from,  source  node  packet 
routing  matrices,  and  how  the  simualtion  is  run  for  the 
Baseline  model. 
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VI  Initial  Baseline  Considerations 

In  order  to  have  something  to  compare  the  algorithm 
version  of  the  simulation  model  against,  a  baseline  or 
control  program  was  developed.  The  basic  processes  of 
packet  generation,  assigning  lengths,  priorities  and  des¬ 
tination  node  IDs  to  packet  parameters  are  the  same  in  both 
programs.  Packet  input  and  output  processing  is  the  same  in 
both  programs  as  is  the  termination  of  packets.  The  only 
difference  in  the  two  programs  is  the  methodology  used  in 
selecting  trunks  and  links  over  which  to  route  the  packets. 

The  baseline  model  uses  static  or  fixed  routing,  that 
is,  given  a  packet  currently  at  node  i  destined  for  node  j, 
a  predetermined  link  k  is  always  used.  This  route  is  derived 
from  a  given  network  routing  matrix  to  be  explained  shortly. 
The  algorithm  model,  of  course,  uses  the  hierarchical  routing 
algorithm  to  determine  the  best  link  and  therefore  the  best 
trunk  over  which  to  route  the  packet.  The  next  chapter 
explains  the  incorporation  of  the  routing  algorithm  into 
the  simulation  model. 

This  chapter  explains  how  the  packet  generation  rate 
is  calculated  for  each  node.  Since  each  node  does  not 
have  the  same  number  of  Host  computers  and  terminals 
connected  to  it,  the  number  of  packets  entering  the  node 
(from  the  Host  computer  and  terminals)  per  unit  of  time  is 
different  for  each  node.  The  procedure  used  to  calculate 
the  packet  mean  interarrival  rate  (time  between  arrivals 


of  packets  to  the  node)  is  explained  in  detail.  The  assign¬ 
ment  of  a  destination  to  a  packet  is  based  on  the  fact  that 
a  certain  percentage  of  the  packets  originating  at  a  node 
goes  to  each  of  the  other  nodes  including  the  originating, 
or  source  node.  These  percentages  and  how  they  are  cal¬ 
culated  for  each  node  are  discussed  along  with  how  the  per¬ 
centages  are  implemented  in  the  baseline  program.  The  ideas 
of  different  packet  priorities  and  accumulation  of  delay 
time  are  also  discussed  in  this  chapter.  The  Network  Routing 
Matrix  (Ref  17 ill),  is  used  to  determine  which  node  processes 
the  packet  next,  given  the  source  node  and  destination  node 
of  the  packet.  The  part  it  plays  in  the  simulation  model  is 
also  explained  in  this  chapter.  The  last  section  of  this 
chapter  explicitly  explains  how  the  simulation  model  was  run 
and  gives  some  sample  results  of  the  baseline  simulation  model. 

Packet  Interarrival  Times  and  Length  Determinations 

The  calculation  for  the  packet  interarrival  times  is  a 
complex  process.  It  involves  the  use  of  predetermined  statis¬ 
tical  data  (Ref  17iV-B-110),  assumptions  made  by  Systems 
Control  Incorporated  (SCI)  during  their  development  and  par¬ 
tial  simulation  of  the  hierarchical  routing  algorithm 
(Ref  18tl-2),  and  a  limited  number  of  educated  assumptions  on 
the  author's  part.  The  author's  assumptions  were  confirmed 
by  telephone  conversations  with  DCA  representatives  through 
the  Defense  Communications  Engineering  Center  (DCEC),  Reston, 
Virginia.  When  used,  each  of  the  assumptions  referenced 

76 


above  is  explained. 

When  a  packet  is  generated,  among  other  parameters,  it 
is  assigned  one  of  two  lengths.  A  long  packet  is  4704  bits 
in  length  and  a  short  packet  is  600  bits  in  length.  The  per¬ 
centage  factors  of  how  many  long  and  short  packets  are  gener¬ 
ated  are  the  other  results  of  the  packet  interarrival  time 
calculations. 

In  their  development  and  partial  simulation  of  the  algo¬ 
rithm,  SCI  calculated  that  81.3#  (Ref  18 «2),  of  all  the  bits 
in  the  network  are  data  packet  bits.  The  maximum  throughput, 
or  trunking  capacity  (maximum  number  of  bits  able  to  be 
supported  by  the  total  number  of  trunks  in  the  network)  of 
the  AUTODIN  II  network  is  simply  the  number  of  trunks  multi¬ 
plied  by  the  transmission  speed  of  the  trunks.  By  the  calcu¬ 
lation  28  x  2  x  56KBPS  (the  2  is  because  each  trunk  is  two- 
way),  the  trunking  capacity  is  3*136  million  bits  per  second 
(MBPS).  The  number  of  data  bits  in  the  network  then  becomes 
81.3#  of  3*136  MBPS  or  2,549»568  data  packet  bits.  A  deter¬ 
mination  has  to  be  made  as  to  how  those  bits  enter  the  network, 
that  is,  how  many  of  those  bits  enter  the  network  from  Albany, 
Andrews,  and  the  rest  of  the  nodes. 

There  are  two  types  of  subscribers  (users)  that  connect 
to  nodes  and  send  data  into  the  network.  They  are  Host  and 
terminal  subscribers.  Host  subscribers  are  large  information 
or  computational  sources  (referred  to  also  as  host  systems 
or  host  computers)  (Ref  17iV-l-55)*  Terminal  subscribers, 
which  include  terminal  computers,  are  generally  mini-  to 

77 


small-size  computers  that  serve  as  communications  handlers 
or  terminal  concentrators  and  may  also  provide  some  limited 
local  computational  capacity  (Ref  I71V-I-55).  They  also  in¬ 
clude  input-output  devices  such  as  tele typewri ter s  (TTYs), 
cathode  ray  tube  (CRT)  terminals,  printers,  remote  batch 
terminals,  card  and  magnetic  tape  terminals  (Ref  17«V-l-56). 
Terminal  users  input  data  that  is  formatted  into  a  combin¬ 
ation  of  long  data  packets  (4?04  bits)  and  short  data  pac¬ 
kets  (600  bits)  (Ref  18 il).  Host  users  input  data  that  is 
formatted  into  long  data  packets  only  (Ref  18 il).  The  per¬ 
centage  of  short  packets  generated  from  terminal  users  is 
81.25#  and  the  percentage  of  long  packets  generated  is 
18.75#  (Ref  18 1 1 ) • 

The  actual  statistics  to  which  the  above  percentages 
apply  is  in  the  form  of  information  provided  by  DCA 
(Ref  I71V-B-IIO)  as  shown  in  Table  II.  Since  the  adjusted 
trunk  capacity  figure  of  2,549,568  bits  is  for  data  packets 
(including  a  network  overhead  of  528  bits  of  network  infor¬ 
mation),  the  size  of  a  long  data  packet  is  4704  +  528  bits, 
or  5232  bits.  The  size  of  a  short  data  packet  thus  becomes 
600  +  528  or  1128  bits.  These  two  numbers  are  used  as  shown 
in  Table  III,  in  order  to  calculate  normalized  figures. 
Normalization  is  necessary  because  the  total  number  of  bits 
input  from  Host  and  terminal  subscribers  is  slightly  less 
than  the  adjusted  trunk  capacity  of  the  network.  An  assump¬ 
tion  on  the  part  of  the  author  is  that  the  figures  shown  in 
Table  III  represent  the  maximum  number  of  bits  input  into  the 
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network  if  the  Host  and  terminal  subscribers  (currently 
scheduled  to  be  part  of  the  AUTODIN  II  netwrok)  input  data 
at  their  respective  maximum  rates.  Theoretically,  if  this 
were  to  occur,  the  entire  network  would  be  in  a  state  of 
saturation.  The  Host  and  terminal  subscribers  input  81.3% 
of  the  trunking  capacity  of  3*136  MBPS  and  the  nodes  them¬ 
selves  add  the  additional  18.7%  due  to  required  network  con¬ 
trol  and  synchronization  information.  The  last  thing  shown 
in  Table  III  is  the  determination  of  the  percentages  of  long 
and  short  packets  (62%  of  all  packets  input  are  long  and  38% 
are  short  packets). 

Table  IV  shows  the  input  packet  rate  per  node  at  the 
maximum  subscriber  input  rate.  Under  normal  operating  con¬ 
ditions,  the  subscribers  would  input  at  about  20%  of  the  rate 
shown.  Thus,  the  network  would  nominally  operate  at  20% 
trunk  loading  as  confirmed  from  DCA  by  telephone  conversation. 
Table  IV  shows  what  the  input  rates,  in  packets  per  second 
(PPS),  into  the  nodes  would  have  to  be  in  order  to  produce 
trunk  loading  of  100%  (saturation),  90%,  80%,  70%,  60%,  50%, 
40%,  and  20%  (normal  operating  traffic  level).  Also  shown 
in  Table  IV  are  the  packet  interarrival  times  (inverse  of 
the  input  packet  rate)  in  milliseconds  per  packet  (MSPP). 

These  constitute  the  mean  packet  interarrival  times  used 
to  generate  packets  in  the  simulation  model  using  the 
GENERATE  blocks.  Appendix  A  is  the  portion  of  the  baseline 
simulation  program  representing  the  code  necessary  for  the 
Andrews  node.  The  code  closely  resembles  the  Block  Diagram 
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TABLE  IV 
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code  in  Figure  21,  Chapter  V.  The  encircled  "AM  shows  the 
GENERATE  block  instruction  with  the  corresponding  packet 
mean  interarrival  time  of  26  milliseconds  exponentially 
distributed  by  the  FUNCTION  EXPON.  Further  explanation  of 
the  use  of  the  different  packet  interarrival  times  is  forth¬ 
coming  in  a  later  section  of  this  chapter. 

Priority  Length  and  Destination  Assignments 

In  order  for  a  packet  to  receive  preferential  treatment 
(be  processed  or  transmitted  before  other  packets  even  though 
it  entered  a  queue  after  the  other  packets)  it  is  flagged  or 
assigned  a  priority  level  higher  than  those  of  the  other  pac¬ 
kets.  In  the  simulation  model  a  packet  is  either  a  priority 
or  a  non-priority  packet.  A  priority  packet  does  not  preempt 
or  interrupt  a  non-priority  packet  that  is  being  processed  by 
an  SCM  or  transmitted  on  a  trunk.  Priority  packets  do  move 
to  the  head  of  the  queue  by  passing  non-priority  packets. 
Ninety  nine  percent  of  the  input  packets  are  non-priority, 
and  one  percent  is  given  the  priority  assignment  (Ref  18 * 3 ) . 
This  process  is  accomplished  at  the  encircled  "B"  in  Appendix 
A.  The  TRANSFER  instruction  sends  1 %  of  the  packets  to  the 
instruction  labeled  HIGH2  where  they  are  given  the  priority 
status  of  2  (stored  in  parameter  2  of  the  packet).  The  other 
99#  of  the  packets  are  given  the  non-priority  status  of  1 
(stored  in  parameter  2  of  the  packet).  The  values  in  para¬ 
meter  2  are  used  later  on  to  determine  the  amount  of  pro¬ 
cessing  time  a  packet  requires. 
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As  determined  in  Table  III,  62 #  of  the  packets  gen¬ 
erated  are  long  packets  and  38#  are  short  packets.  The 
simulation  model  accomplishes  this  task  at  the  encircled 
"C"  in  Appendix  A,  The  TRANSFER  instruction  labeled  C0N21 
sends  62#  of  the  packets  to  label  L0NG2  where  parameter  3 
of  those  packets  are  given  the  value  of  2  denoting  a  long 
packet.  The  other  38#  of  the  packets  fall  through  TRANSFER 
instruction  and  are  given  the  value  of  1  in  parameter  3 
denoting  a  short  packet.  The  values  assigned  to  parameter  3 
of  each  packet  are  used  later  on  to  determine  the  amount  of 
transmission  time  each  packet  requires. 

In  determinimg  which  node  is  to  be  the  destination  of  a 
packet,  the  information  (Ref  l?tV-B-131)  in  Table  V  is  used. 
It  represents  node-to-node  data  packets  exchanged  for  a  given 
time  unit.  Each  row  represents  the  number  of  packets  term¬ 
inating  locally  (source  node  and  destination  node  the  same) 
and  the  number  of  packets  the  source  node  sends  to  all  the 
other  nodes.  From  this  information  Table  VI  is  derived. 
Instead  of  the  number  of  packets  sent  to  each  of  the  other 
nodes,  entries  in  Table  VI  represent  the  ratios  of  packets 
transmitted  to  each  of  the  other  nodes  from  a  source  node. 

The  ratios  are  calculated,  using  Table  V,  by  summing  up 
each  row  to  obtain  the  total  number  of  packets  sent  by  a 
source  node.  Each  column  entry,  given  the  source  node,  is 
divided  by  the  appropriate  row  total  to  calculate  the  ratio 
of  packets  sent  to  a  particular  destination  node.  At  the 
encircled  "D"  in  Appendix  A,  parameter  1  of  a  packet  is 
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TABLE  V 


assigned  a  destination  node  ID  number  using  the  FUNCTION 
DEST2.  The  function,  encircled  "E"  in  Appendix  A,  uses 
random  number  generator  two,  RN2,  to  generate  a  random 
number  from  .00000  to  1.00000.  The  next  line  following 
DEST2  contains  eight  discrete  intervals,  each  interval 
separated  by  a  slash.  The  decimal  numbers  in  the  intervals 
are  the  cumulative  sums  of  Andrew* s  row  in  Table  VI.  The 
difference  between  the  nth  and  nth  +  1  intervals  are  the 
ratio  entries  in  Andrew* s  row  of  Table  VI.  For  example, 
the  difference  between  the  4  th  and  5^  entries  is  .7777  - 
.6592,  or  .II85.  Looking  at  Table  VI,  this  number  corres¬ 
ponds  to  the  ratio  of  packets  originating  at  Andrews  with 
Hancock  as  the  destination  node.  After  the  .7777  entry  is 
the  number  5,  which  is  Hancock’s  node  ID  number.  If  the  gen¬ 
erated  random  number  is  between  .6 592  and  .7777»  then  para¬ 
meter  1  of  the  packet  currently  being  processed  is  assigned 
the  integer  number  5»  denoting  the  destination  node  of  the 
packet  to  be  Hancock.  The  rest  of  the  intervals  in  DEST2 
are  derived  in  the  same  fashion,  that  is,  for  random  num¬ 
ber  values  in  the  range  .00000  to  .11940,  the  packet  is  assign¬ 
ed  destination  node  ID  of  1  (Albany),  .11941  to  .54630 
(.1194  +  .4269)  the  ID  is  2  (Andrews),  .54631  to  .5893 
(.5463  .0430)  the  ID  is  3  (Ft.  Detrick),  and  so  on. 

All  of  the  pertinent  packet  parameter  values  have  now 
been  assigned  to  the  packet.  Parameter  1  contains  the  des¬ 
tination  ID  number  (1  to  8),  parameter  2  contains  the  priority 
status  (1  -  non-priority,  2  -  priority),  parameter  3  con- 


tains  the  packet  length  (1  -  short,  2  -  long),  and  parameter 
4  contains  the  number  2  which  indicates  Andrews  as  the  pac¬ 
kets  source  node.  The  packet  is  now  ready  to  enter  one  of 
the  two  SCMs  to  be  processed  and  transmitted  on  to  the  next 
node  (which  might  be  its  destination  node  or  a  tandem  node). 
This  is  accomplished  by  the  TRANSFER  instruction  (encircled 
"F"  in  Appendix  A) . 

Packet  Processing  and  Transmission 

Not  only  do  new  packets  just  generated  enter  either  ANDY1 
or  ANDY2,  but  also  the  packets  sent  to  the  Andrews  node  from 
other  nodes  enter  Andrews  at  these  two  entry  points.  For 
ease  of  explanation,  assume  the  packet  is  sent  to  SCM  1.  In 
order  to  receive  service,  the  packet  enters  the  processor's 
input  queue,  seizes  the  processor  when  it  is  available,  and 
departs  the  processor's  input  queue  as  shown  in  Appendix  A, 
encircled  "G".  If  the  packet  is  a  priority  packet,  it  moves 
to  the  front  of  the  FIRST-IN-FIRST-OUT  (FIFO)  input  queue 
ANQU1.  If  the  processor  ANPS1  is  busy  (currently  seized  by 
another  packet),  the  queued  packet  remains  in  the  queue  until 
the  processor  is  free.  When  the  packet  seizes  the  processor 
it  acquires  a  certain  amount  of  processing  delay  time  depend¬ 
ing  on  its  priority  (encircled  "H",  Appendix  A).  To  deter¬ 
mine  the  priority  of  a  packet,  FUNCTION  PRI21  (encircled  "I") 
is  used  by  the  TRANSFER  instruction.  Packet  parameter  2  is 
inspected  and  if  it  is  a  1  (non-priority),  the  packet  is 
transferred  to  the  label  L0P21  where  the  ADVANCE  instruction 
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adds  5  milliseconds  (Ref  17iV-B-26)  to  the  packet's  system 
time  parameter  (located  in  GPSS's  system  tables).  If  the 
packet  parameter  is  a  2  (priority),  the  packet  transfers  to 
the  label  HIP21  where  the  ADVANCE  instruction  adds  3  milli¬ 
seconds  (Ref  17*V-B-26)  to  the  packet's  system  time  para¬ 
meter.  After  receiving  the  appropriate  amount  of  SCM  pro¬ 
cessing  time,  the  processor  is  released  (encircled  "J", 
Appendix  A) . 

The  packet  now  enters  the  SAVEVALUE  instructions  (en¬ 
circled  "K",  Appendix  A).  ANQll  through  ANQ42  are  the  out¬ 
put  queues  associated  with  the  output  trunks  emanating  from 
the  Andrews  node.  The  lengths  of  the  queues,  or  number  of 
packets  in  each  queue,  are  stored  in  SAVEVALUEs  8  through  13. 
All  the  necessary  conditions  for  the  packet  to  start  the 
transmission  process  have  now  been  accomplished. 

When  the  packet  enters  the  TRANSFER  instruction  loc¬ 
ated  at  the  encircled  "L",  Appendix  A,  the  FUNCTION  R0UT2 
(encircled  "M" ,  Appendix  A)  uses  the  value  in  packet  para¬ 
meter  1  to  determine  which  outgoing  link  on  which  to  queue 
the  packet.  When  the  destination  node  of  a  packet  is  not  a 
directly  connected  node  to  the  node  processing  the  packet,  a 
tandem  or  intermediate  node  is  selected.  Due  to  the  2-hop 
constraint,  if  a  node  is  a  tandem  node  it  is  directly  con¬ 
nected  to  the  destination  node.  Table  VII  (Ref  18 ill)  is 
used  in  the  next-node  determination.  Given  the  source  node 
(row  headings),  and  the  destination  node  (column  headings), 
the  entries  give  the  next  node  to  process  the  packet  (inc- 
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ludins  tandem  and  destination  nodes).  The  values  of  the 
FUNCTION  R0UT2  correspond  to  Andrews'  row  of  Table  VII.  The 
link  labels  in  R0UT2  are  the  links  to  route  packets  on  to 
get  to  the  node  given  in  Andrews'  row.  For  example,  packets 
destined  for  Gentile  (ID  of  4)  from  Andrews  pass  through 
Ft.  De trick.  Where  Andrews'  row  and  Gentile's  column  inter¬ 
sect  (in  Table  VII),  Ft.  Detrick  is  the  node  given  to  process 
the  packet  next.  For  Gentile  being  the  packet's  destination, 
parameter  l's  value  is  a  4.  The  4th  entry  in  R0UT2,  each  en¬ 
try  separated  by  a  slash,  gives  the  label  ANQL2  which  is  the 
link  connecting  Andrews  and  Ft.  Detrick.  If  the  value  is  a 
1,  the  packet  transfers  to  label  ANK11 ,  or  if  it  is  a  2,  the 
packet  transfers  to  the  label  TERM2  for  local  "delivery" 
(termination).  As  seen  in  the  rest  of  the  function,  for 
each  destination  ID  number  (1  to  8),  a  label  exists  where 
the  packet  is  transferred  to.  In  the  event  a  link  contains 
more  than  one  trunk,  as  in  the  link  between  Andrews  and 
Ft.  Detrick,  ANQL2  (encircled  "N",  Appendix  A),  the  individual 
trunk  queue  lengths  are  compared  to  queue  the  packet  on  the 
shorter  of  the  two.  If  the  packet  is  to  be  sent  to  Ft.  Det- 
rick,  the  output  queue  lengths  (ANQ21  and  ANQ22)  are  com¬ 
pared.  If  the  length  of  ANQ21  is  shorter  than  ANQ22,  the 
packet  is  sent  to  label  ANK21  (primary  trunk  to  Ft.  De trick) 
otherwise  it  is  sent  to  label  ANK22  (secondary  trunk  to 
Ft.  Detrick).  Since  the  link  to  Tinker  (ANQL4)  contains 
two  trunks,  a  similar  process  is  used  for  trunk  selection. 

For  packets  going  to  Albany  or  Hancock,  they  are  sent  directly 


to  the  respective  trunks  (ANK11  or  ANK31 ) • 

Locally  terminating  packets  are  sent  to  the  label  TERM2 
(encircled  "P",  Appendix  A)  where  the  TABULATE  instruction 
enters  the  packet's  accumulated  delay  time  (value  in  the  pac¬ 
ket's  system  time  Parameter  Ml)  in  a  table  labeled  RTIME  (de¬ 
fined  encircled  "Q",  Appendix  A).  The  Table  RTIME  defines 
the  system  parameter  to  be  tabulated  (Ml),  the  first  boundary 
point  (20),  width  of  each  intermediate  table  in:rrval  (20), 
and  the  total  number  of  intervals  in  the  table  (^00).  For 
example,  if  a  packet  had  accumulated  15  milliseconds  of  delay 
time,  it  would  be  tabulated  in  the  first  interval  0-20.  If 
another  packet  has  amassed  a  total  of  93  milliseconds  (msec) 
of  delay  time,  it  would  be  tabulated  in  the  table's  interval 
80  -  100.  Table  VIII  is  an  example  of  a  sample  output  of 
system  table  RTIME.  The  UPPER  LIMIT  column  shows  the  upper 
limit  of  the  intervals,  that  is,  an  upper  limit  of  20  repre¬ 
sents  the  interval  0-20  msec,  60  represents  the  interval 
41  -  60  and  so  on.  The  OBSERVED  FREQUENCY  column  gives  the 
total  number  of  packets  whose  delay  times  Eire  within  the  cor¬ 
responding  time  interval.  The  other  entries  are  self-explan¬ 
atory.  After  the  packet's  delay  time  is  tabulated  in  RTIME, 
the  program  tabulates  the  packet's  delay  time  in  other  tables 
depending  on  the  packet's  priority  and  length  as  shown  in  the 
section  of  code  at  the  encircled  "R",  Appendix  A.  The  packet 
finally  leaves  the  network  by  entering  one  of  the  TERMINATE 
instructions. 

For  non-priority  packets,  assume  the  packet's  parameter 
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Example  Output  of  System  Delay  Table  RTIME 
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TABLE  VIII  (CONT'D) 
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All  remaining  classes  are  zero* 


1  contains  the  value  of  8  (Tinker's  ID).  Assume  also  af¬ 
ter  comparing  trunk  queue  lengths,  the  packet  transfers 
to  ANK42  (encircled  "S",  Appendix  A).  The  packet  enters 
queue  ANQ42  which  corresponds  to  the  4th  link,  2nd  trunk 
queue.  When  the  trunk  ANTK5  is  available,  the  packet 
seizes  it  and  departs  ANQ42.  Depending  on  the  packet's 
parameter  2  value,  the  packet  is  transferred  to  ANS42 ,  if 
a  short  packet,  to  acquire  2?  msec  transmission  time,  or 
to  ANL42,  if  a  long  packet,  to  acquire  100  msec.  The 
transmission  times  of  2?  and  100  msec  are  derived  by  di¬ 
viding  the  packet's  length  by  the  transmission  speed  of  the 
trunk  (56000  KBPS)  plus  a  propagation  factor  of  7  msec. 

After  acquiring  the  appropriate  transmission  time,  the  pac¬ 
ket  releases  ANTK5  and  transfers  to  Tinker's  SCM  3»  TNKR3, 
for  further  processing. 

Running  the  Baseline  Model 

In  any  simulation,  the  first  question  to  ask  is  "What 
exactly  do  I  want  the  simulation  model  to  show?"  In  the 
baseline  simulation,  and  in  the  algorithm  simulation,  there 
are  several  items  of  interest  such  as  mean  packet  delay, 
maximum  packet  delay,  and  adaptability  (reaction  characteris¬ 
tics)  of  the  routing  discipline  to  topological  changes  and 
traffic  level  fluctuations.  The  purpose  of  this  section  is 
to  explain  exactly  how  the  baseline  simulation  model  is  run 
on  the  CDC  CYBER  1?5  computer  in  order  to  show  the  items  of 
interest  detailed  above.  Due  to  time  and  resource  constraints 
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only  the  traffic  level  is  allowed  to  fluctuate.  In  the 
next  chapter,  the  incorporation  of  the  hierarchical  rout¬ 
ing  algorithm  is  explained  and  both  topological  changes 
and  traffic  level  fluctuations  are  simulated.  The  same 
procedure  used  in  the  baseline  program  to  vary  the  traffic 
levels  holds  for  the  algorithm  simulation  model  also. 

Basically,  the  model  is  run  for  five  seconds  at  20% 
saturation  in  order  to  establish  a  steady-state  system. 

The  program  is  then  allowed  to  run  for  thirty  seconds, 
again  at  a  20 %  saturation.  Thirty-five  seconds  into  the 
simualtion,  a  traffic  volume  spike  occurs  for  one  second, 
then  the  traffic  volume  returns  to  the  normal  operating 
level  of  20%  of  saturation  and  remains  there  for  twenty  more 
seconds.  Four  runs  are  made  with  the  spiked  traffic  level 
being  60%,  70$,  80$,  and  90%  of  network  saturation.  For 
each  of  the  four  runs  statistics  are  output  after  thirty 
second  continuous  level  of  20%  saturation,  after  the  one 
second  burst,  and  once  each  second  thereafter  for  the  next 
twenty  seconds.  The  reasoning  behind  so  many  statistic 
dumps  is  to  show  how  the  routing  cPiscipline  (static  in  the 
baseline  model,  adaptive  routing  in  the  algorithm  model) 
behaves  after  the  one  second  traffic  burst. 

The  operating  system  of  the  CYBER  175  allows  remote 
terminals  (CRTs  and  Texas  Instrument's  Silent-700  terminals) 
to  access  the  main  computer  through  a  portion  of  the  operating 
system  called  Intercom.  Using  a  normal  LOGIN  procedure,  the 
interactive  user  can  create,  manipulate,  and  save  files 
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(usually  programs)  once  into  the  Intercom  portion  of  the 
operating  system.  The  GPSS  code  for  the  baseline  program 
was  entered  similarly  and  saved  on  permanent  file.  To 
make  a  set  of  computer  runs,  a  copy  of  the  baseline  program 
is  called  into  a  file  editor  and  made  a  local  file  (local 
as  long  as  the  Intercom  session  lasts)  with  the  file  name  of 
say,  DIN. 

For  a  set  of  four  runs,  the  only  section  of  code  altered 
is  the  section  denoted  at  the  encircled  "T",  Appendix  A. 

GPSS  allows  redefinition,  or  overlaying,  of  pre¬ 
viously  labeled  instructions  with  new  instructions  labeled 
with  the  same  label  name.  For  the  first  run,  (spiked  traffic 
at  60%  saturation),  the  code  denoted  by  the  encircled  "U"  is 
deleted  from  the  edit  buffer.  The  rest  of  the  edit  buffer 
which  not  only  contains  the  code  as  shown  in  Appendix  A  for 
the  Andrews  node  but  also  similar  code  for  each  of  the  other 
seven  nodes  in  the  network,  is  saved  under  another  local  file 
name  (say,  RUN1).  Appropriate  Intercom  control  commands  are 
entered  through  the  keyboard  to  start  the  compilation  and 
execution  of  the  program  with  the  file  name  of  RUN1.  The  file 
with  the  name  DIN  is  then  called  back  into  the  edit  buffer  to 
be  modified  for  the  second  run.  This  time  the  section  of 
code  denoted  by  the  encircled  "V"  is  deleted  from  the  edit 
buffer.  The  rest  of  the  edit  buffer  is  saved  under  a  new 
local  file  name  (say,  RUN2) •  Again  appropriate  Intercom  con¬ 
trol  commands  are  entered  to  start  the  compilation  and  exe¬ 
cution  of  the  program  with  the  file  name  of  RUN2.  Similarly, 
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for  the  3rd  and  4th  runs  DIN  is  called  back  into  the  edit 
buffer  and  changed  appropriately  following  the  same  pro¬ 
cedure  as  outlined  above  but  removing  only  those  sections 
of  code  not  relating  to  spiked  traffic  level  of  interest. 

During  execution,  the  time-controlling  portion  of 
code  is  denoted  by  the  encircled  "W",  Appendix  A,  and  each 
of  the  remaining  START  instructions.  Each  time  unit  in  the 
simulation  program  is  considered  to  be  one  millisecond  (msec). 
Thus  the  GENERATE  1000  instruction  generates  one  transaction 
per  1000  msecs  or  one  second.  The  first  operand  of  a  START 
instruction  is  stored  by  the  GPSS  simulation  control  pro¬ 
cessor  and  acts  like  a  time-control  counter.  After  a  second 
of  simulation  time,  the  GENERATE  1000  instruction  generates 
a  transaction  which  immediately  enters  the  TERMINATE  1  ins¬ 
truction.  The  "1"  of  the  TERMINATE  instruction  subtracts 
one  from  the  time-control  counter.  If  the  counter's  value 
is  aero  that  portion  of  the  simulation  is  ended.  If  it  is 
not  zero,  the  simulation  runs  for  another  second,  the 
GENERATE  1000  instruction  generates  another  transaction  which 
immediately  enters  the  TERMINATE  1  instruction.  One  is  sub¬ 
tracted  from  the  time-control  counter.  If  the  counter  is 
zero,  the  simulation  ends,  if  not  zero,  the  simulation  runs 
for  another  second.  When  the  counter  reaches  zero,  the  GPSS 
simulation  processor  scans  the  following  instructions  until 
it  finds  another  START  instruction  or  the  END  instruction. 

If  the  simulation  processor  finds  labeled  instructions  be¬ 
tween  START  instructions,  the  processor  overlays  the  existing 
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instructions  with  the  same  labels  in  the  main  portion  of  the 
program  with  the  new  instructions.  This  is  how  the  traffic 
intensity  levels  are  changed  in  the  simulation.  The  different 
mean  interarrival  times  in  the  labeled  GENERATE  instructions 
come  from  Table  IV.  When  the  END  instruction  is  encountered, 
the  simulation  is  ended. 

In  words,  the  execution  of  the  program  (for  the  60# 
spike)  is  as  follows i  At  the  start  of  the  execution,  the  sim¬ 
ulation  clock  is  set  to  zero.  Execution  begins  and  packets 
are  generated  in  the  main  body  of  the  program  moving  through 
the  network  code  as  instructed.  Also  at  time  0,  the  GPSS 
control  processor  scans  down  to  the  first  START  instruction 
(START  5,  NP)  where  the  NP  stands  for  no  statistics  print¬ 
out  after  this  section  of  the  simulation  ends.  The  time- 
control  counter  is  set  to  5  and  execution  actually  starts. 
After  one  second,  the  GENERATE  1000  instruction  (encircled 
"W"),  generates  a  transaction  which  immediately  enters  the 
TERMINATE  1  instruction.  One  is  subtracted  from  the  time- 
control  counter.  If  non-zero,  simulation  continues,  if  zero, 
statistics  are  output  and  the  GPSS  processor  scans  down  to 
the  next  START  instruction  (START  30,, 30).  The  RESET  ins¬ 
truction  clears  all  the  statistics  accumulated  except  for 
those  involving  facilities  (SCM  processors  and  the  trunks) 
and  queues  (SCM  processor  input  queues  and  the  trunk  queues). 
The  first  "30"  of  the  START  instruction  sets  the  time-control 
counter  to  30.  The  second  "30"  informs  the  GPSS  processor  to 
generate  output  statistics  after  30  transaction  terminations 
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(30  seconds).  After  30  more  seconds  of  simulation  time,  the 
statistics  requested  are  output  and  the  GPSS  control  pro¬ 
cessor  scans  down  to  the  next  START  card  (START  01,, 01).  The 
instructions  labeled  ALBAN  through  TINKE  in  the  selection  of 
code  denoted  by  the  encircled  "X" ,  Appendix  A,  replaces  the 
instructions  under  the  same  label  name  in  the  main  program. 

The  packet  generating  instruction  in  each  of  the  node's 
code  is  modified  to  generate  packets  at  60%  saturation  rate. 
The  RESET  instruction  clears  all  the  statistics  except  for 
those  given  in  the  instruction.  The  modified  program  runs 
for  one  second,  terminates  and  statistics  are  output.  The 
GPSS  control  processor  scans  down  to  the  next  START  instruc¬ 
tion  (START  1,,1).  The  instructions  denoted  by  the  encirc¬ 
led  "Y"  are  used  to  reestablish  the  original  packet  generation 
intensity  of  20 %  saturation.  The  statistics  are  reset  and 
execution  begins  again.  This  procedure  occurs  again  and  again 
until  the  END  instruction  is  encountered  and  the  complete  sim¬ 
ulation  ends.  It  should  be  noted  that  when  a  time  interval 
elapses,  the  status  of  the  model,  that  is,  the  number  of 
packets  in  the  queues  and  packet  positions  throughout  the 
model  does  not  change.  The  RESET  instruction  clears  only 
the  appropriate  tables  (statistics). 

Baseline  Results 

The  results  of  the  baseline  simulation  are  of  no  real 
importance  except  to  get  a  feel  of  how  the  basic  model  op¬ 
erates.  Items  of  interest  are  the  mean  packet  delay,  maxi¬ 
mum  packet  delay  and  (to  see)  how  these  values  change  when 
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the  traffic  intensity  increases.  In  the  next  chapter  these 
results  are  compared  against  similar  algorithm  model  runs 
to  get  a  feel  of  the  algorithm  operation. 

The  test  way  to  show  the  results  is  in  the  form  of 
graphs.  Specific  data  relative  to  the  above  mentioned  items 
of  interest  are  extracted  manually  from  the  simulation  statis¬ 
tic  outputs  and  put  together  to  form  five  illustrations,  three 
graphs  and  two  tables.  The  statistical  output  data  starts  at 
the  30  second  time  interval.  The  mean  delay  value  of  the  30 
second  time  mark  (in  parenthesis  on  the  graphs)  is  the  mean 
packet  delay  over  a  30  second  simulation  time  period  during 
which  the  system  has  reached  equilibrium.  Figure  22  shows 
the  graphical  results  of  data  taken  from  the  system  delay 
table,  RTIME,  for  each  of  the  four  simulation  runs.  Also 
shown  in  Figure  22  are  the  significant  maximum  packet  delay 
times  encountered  for  each  run  (in  parentheses  on  the  graph). 
These  values  indicate  that  during  the  time  interval  in  which 
the  statistics  were  taken,  a  packet  terminated  having  the 
delay  time  indicated  in  parentheses.  Figure  23  shows  the 
results  of  data  taken  from  the  non-priority  -  short  packet 
delay  table,  RTLOS ,  along  with  the  significant  maximum  delay 
times.  Figure  24  shows  the  results  of  data  taken  from  the 
non-priority  -  long  packet  delay  table,  RTLOL,  and  the 
associated  significant  delay  times*  Since  only  1 %  of  the 
packets  generated  as  priority  packets,  the  data  on  the 
priority  short  and  long  packets  are  not  completely  repre¬ 
sentative*  The  number  of  priority  packets  generated  in  the 
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PACKET  DELAY 
(MSEC) 


PACKET  DELAY 
(MSEC) 


one  second  time  intervals  is  so  small  as  to  infer  incorrect 
mean  delay  times  if  shown  in  graphical  form.  During  some 
of  the  one  second  time  intervals,  only  one  priority  -  short 
packet  is  generated  and  in  the  other  one  second  time  inter¬ 
vals  only  one  priority  -  long  packet  is  generated.  In  some 
of  the  one  second  time  intervals,  no  priority  packets  were 
generated  at  all l  The  best  way  to  show  these  types  of  results 
is  in  tabular  format.  Table  IX  contains  data  extracted  from 
the  statistical  outputs  for  priority  -  3hort  packets,  and 
Table  X  shows  the  data  extracted  for  priority  -  long  packets. 

Figure  25  contains  information  on  packet  delays  extracted 
from  DC A' s  System  Performance  Specifications  (Ref  19)  for  the 
AUTODIN  II  network. 

Category  Category  I  Category  I  II 

Type  (Priority)  III  IV 

(Non-priority) 


Mean 
(msec ) 

Max 
(msec ) 

Mean 
(msec ) 

Max 

(msec) 

Short  Packets 
(1120  bits) 

180 

580 

380 

1350 

Long  Packets 
(5224  bits) 

330 

730 

530 

1500 

Pig  25 •  AUTODIN  II  Packet  Delay  Specifications 


The  delays  shown  axe  for  the  two-hop  case,  that  is, 
packets  are  assumed  to  traverse  a  source,  tandem,  and  des¬ 
tination  node,  and  SCM  processors  are  operating  at  eighty 
per  cent  utilization  (Ref  19i43).  The  maximum  is  defined  to 
be  the  99th  percentile  of  delivery  delay  distribution 
(Ref  19«43),  that  is,  one  tenth  of  one  per  cent  of  the  packets 
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System  Delay  (priority  -  short  packets) 
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System  Delay  (priority  -  long  packets) 
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in  the  system  are  allowed  to  incur  a  delay  not  to  exceed 
the  maximum  values  given  in  Figure  25 • 

In  the  steady-state  condition  (the  first  30  seconds 
of  the  simulation),  the  delay  times  accorded  packets  are 
well  within  the  specifications  given  in  Figure  25*  However, 
when  traffic  intensity  spikes  occur,  specifically  the  80 % 
and  90$  spikes,  both  mean  and  maximum  delay  values  are 
exceeded  (in  the  time  frames  immediately  after  the  spike). 

The  DCA  Performance  Specifications  do  not  list  any  concrete 
behavioral  characteristics  the  routing  algorithm  should 
provide  other  than  it  should  recognize  network  degradations 
and  react  efficiently  to  compensate  for  traffic  volume 
fluctuations.  As  seen  in  the  graphs,  the  baseline  static 
routing  discipline  smoothes  back  out  to  an  average  delay 
value  of  about  100  milliseconds  within  six  seconds  after 
the  spike  occurs.  This  six  second  time  span  may  or  may  not 
be  acceptable  considering  this  simulation  is  only  concerned 
with  a  control  simulation  model. 

These  results,  performance  characteristics  and  the  values 
given  in  Figure  25  take  on  a  more  significant  meaning  in  the 
next  chapter.  The  whole  purpose  of  this  paper  is  shown  in 
the  next  chapter  which  explains  how  the  hierarchical  routing 
algorithm  is  incorporated  into  the  baseline  simulation  model. 
The  algorithm  simulation  program  illustrates  the  adaptability 
of  the  algorithm  by  varying  the  traffic  intensities  while 
topological  changes  occur. 
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VII  Algorithm  Description 


As  stated  in  the  previous  chapter,  the  basic  model  and 
the  simulation  code  of  the  baseline  program  remains  the 
same  for  the  algorithm  version  of  the  program.  The 
differences  are  the  addition  of  Fortran  subroutines  rep¬ 
resenting  the  routing  algorithm  and  the  addition  of  GPSS 
instructions  necessary  to  activate  the  Fortran  subroutines 
as  well  as  the  GPSS  code  necessary  to  pass  numerical  data 
to  the  subroutines.  Since  the  algorithm  "Background  Func¬ 
tions"  require  periodic  entry  into  some  of  the  subroutines, 
additional  simulation  code  is  added  to  facilitate  these 
periodic  entries. 

This  chapter  explains  how  the  Fortran  versions  of 
the  algorithm's  modules  are  written  and  how  they  merge 
with  the  baseline  program.  The  GPSS  changes  necessary 
to  accomplish  and  support  the  new  program  are  also  ex¬ 
plained.  In  addition,  the  chapter  explains  exactly  how 
the  new  program  is  run  with  respect  to  normal  network 
configuration,  that  is,  all  nodes,  links,  and  trunks  op¬ 
erational.  The  final  section  of  this  program  explains  how 
pre-selected  trunks,  links,  and  nodes  are  removed  from  the 
network  and  how  the  algorithm  compensates  for  the  degradation. 

Fortran  Subroutines  for  Modules 

Each  module  of  the  algorithm,  MINHOP,  RTABLE ,  PAKROUTE, 
MESGEN,  and  UPDATE,  is  coded  in  Fortran  with  each  node  of 
the  network  maintaining  its  own  copy  of  the  algorithm.  Each 
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module  has  its  own  Fortran  subroutine  counterpart  except 
MINHOP.  There  are  four  subroutines  composing  MINHOPi 
INITIALIZE  (same  as  CONNECTIVITY  described  in  Chapter  IV), 
INITIALIZER,  COMPUTR,  and  CONNRHANGE.  Appendix  B  rep¬ 
resents  Ft.  Detrick's  version  of  the  algorithm.  The  module 
names  and  their  corresponding  subroutine  names  are  the 
followingi 

(1)  MINHOP 

INITIALIZE  -  DEINIT 

INITIALIZER  -  DEINID 

COMPUTER  -  DECOMD 

CONN  CHANGE  -  DECCHG 


(2) 

RTABLE 

-  DERTBL 

(3) 

PAKROUTE 

-  DEPKRT 

(4) 

MESGEN 

-  DEMSGN 

(5) 

UPDATE 

-  DEUPDT 

The  COMMON/DETABL/  statement,  and  the  following 
INTEGER  and  REAL  statements  denoted  by  the  encircled  "A"  in 
Appendix  B  contain  the  algorithm  database.  The  block  is 
in  common  with  each  subroutine  since  each  subroutine  uses 
and  modifies  the  algorithm's  tables,  arrays,  variables  and 
constants  contained  in  the  block.  The  names  used  are  as 
close  as  possible  to  the  names  used  in  the  module  flow¬ 
charts  in  Chapter  IV.  Each  name  is  prefixed  with  "DE" 
to  denote  Ft.  Detrick.  Each  of  the  other  nodes  has  almost 
the  identical  code  written  for  them.  What  differentiates 
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one  node's  code  from  the  other  is  the  first  two  letters 
in  each  name.  Albany's  code  is  prefixed  with  the  letters 
“A L",  Andrews  "AN",  Gentile  "GE",  Hancock  "HA",  McClellan 
"MC",  Norton  "NO",  and  Tinker  "TI".  Other  differences  are 
the  arrays  XXLINS  and  XXBFID.  The  values  of  the  XXLINS 
array  (DELINS  for  Ft.  Detrick  -  encircled  "B"  of  Appendix  B), 
are  the  numbers  of  trunks  within  a  particular  link  indexed 
by  that  link.  For  example,  link  number  1  contains  3  trunks 
and  link  number  4  contains  1  trunk.  Since  each  node  does 
not  have  the  same  number  of  links  or  trunks,  the  array  is 
different  for  each  node.  The  values  of  the  other  array, 

XXBFID  (DEBFID  for  Ft.  Detrick  -  encircled  "C"  of  Appendix 
B) ,  are  the  trunk  numbers  of  the  first  trunk  in  a  particu¬ 
lar  link  indexed  by  that  link.  For  example,  the  first 
trunk  in  link  2  is  trunk  number  4  and  the  first  trunk  in 
link  number  4  is  trunk  number  9*  The  last  two  differences 
are  the  local  node's  ID  number  (DESELF  =  3  Tor  Ft.  De trick  - 
encircled  "D"  of  Appendix  B),  and  the  node  saturation 
threshold  XXT2  (DET2  for  Ft.  Detrick  -  encircled  "E"  of 
Appendix  B).  Each  node  does  not  have  the  same  number  of  SCM 
processors.  Hence  the  number  of  packets  on  input  queue 
for  a  one-processor  putting  it  into  saturation  would  not 
put  a  three-processor  node  into  saturation  since  its  in¬ 
put  queue  can,  in  effect,  hold  three  times  the  number  of  pac¬ 
kets.  Thus  the  saturation  threshold  for  one-processor 
nodes  is  lower  than  that  of  the  three-processor  node.  For 
single  processor  nodes  the  threshold  is  12,  for  dual-processor 
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nodes  it  is  18,  and  for  triple-processor  nodes  the  threshold 
is  20.  For  a  more  detailed  definition  of  the  names  used  in 
the  subroutines  see  Appendix  C.  The  names  in  Appendix  C  are 
for  the  Ft.  Detrick  node  but  can  be  extended  to  the  other 
nodes  by  adding  the  appropriate  prefix. 

The  arguments  of  the  subroutine  call  and  their  declar¬ 
ations  (encircled  "F"  of  Appendix  B)  are  required,  in  the 
order  shown,  by  GPSS.  In  the  main  GPSS  simulation  program 
the  packet's  source  node  ID,  destination  node  ID,  priority 
status  and  length  are  stored  in  parameters  associated  with 

that  packet.  Also  stored  in  the  main  program  (in  the 

r 

r  SAVEVALUEs),  are  the  output  trunk  queue  lengths.  The  values 

are  passed  to  the  Fortran  subroutine  via  the  arguments  of 
the  subroutine  call.  The  packet's  parameters  are  passed 
through  the  named  PARMVA  and  the  queue  lengths  are  passed 
through  the  name  SAVVXX  (SAVVDE  for  Ft.  Detrick).  More 
explanation  of  the  passed  values  is  given  in  the  next  section 
of  this  chapter. 

After  the  64  (8  nodes  of  8  subroutines  each)  subroutines 
are  written,  they  are  compiled  as  one  large  program  and  a 
single  relocatable  binary  file  produced.  The  filename  is  an 
input  parameter  for  the  command  to  execute  the  GPSS  simulation 
program.  Before  execution  starts,  the  binary  file  is  loaded 
into  the  core  and  the  subroutine  names  compared  against  all 
the  subroutine  calls  in  the  main  simulation. 
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Simulation  Program  Modifications 

GPSS  HELP  Blocks.  The  instructions  GPSS  provides  to 
communicate  with  Fortran  subroutines  are  called  HELP  ins¬ 
tructions.  Their  format  is  HELPX  A,  B,  C,  D,  E,  F,  G, 
where  X  is  one  of  A,  B,  or  C,  and  A  through  G  are  the  op¬ 
erands.  Operand  A  is  the  name  of  the  subroutine  and  B 
through  G  are  the  values  passed.  HELPA  instructions  pro¬ 
vide  one-way  communication  to  Fortran  subroutines,  that  is, 
the  values  passed  to  the  subroutine  can  only  be  read.  HELPB 
instructions  provide  for  two-way  communications.  The  values 
passed  are  the  contents  of  the  SAVEVALUEs  explicitly  given 
in  the  operands  B  through  G  which  can  be  read  or  changed 
in  the  subroutine.  HELPC  is  a  very  special  instruction. 

Not  only  does  it  provide  two-way  communication  but  it  also 
makes  all  the  GPSS  internal  system  tables  and  arrays  plus  all 
the  attributes  concerning  queues,  facilities,  SAVEVALUEs,  and 
user  defined  TABLES  available  for  use  and  modification  by  the 
Fortran  subroutine.  The  B-G  operands  are  read-only  but  the 
system  values  can  be  read  and/or  modified.  The  system  values 
are  not  passed  as  arguments  in  the  subroutine  call  but  rather 
made  accessible  to  the  subroutine  (see  encircled  "F"  in 
Appendix  B).  The  order  of  definition  in  the  subroutine  must 
be  the  same  as  shown  in  the  encircled  ”E”  portion  because 
this  order  is  the  way  the  internal  tables,  arrays,  etc.  are 
laid  out  in  core.  Any  scrambling  of  the  order  results  in 
chaos  if  the  values  are  referenced  then  modified.  There  is 
no  control  program  checking  to  ensure  proper  referencing  and 
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usage  of  the  system  values. 

Algorithm  Activation.  Appendix  D  is  the  complete  GPSS 
simulation  program  which  incorporates  the  hierarchical 
routing  algorithm.  There  are  essentially  seven  sections  to 
the  program.  Sections  1  and  7  make  up  the  original  baseline 
program  with  minor  changes  and  is  called  the  main  program. 
Sections  2  and  3  provide  the  necessary  periodic  entry  in  the 
MESGEN  subroutine  at  each  of  the  nodes.  Section  2  handles 
the  entry  into  MESGEN  on  the  FTICK  (to  check  "Bad  News")  in¬ 
terval,  every  100  milliseconds,  and  Section  3  handles  the 
entry  into  MESGEN  on  the  STICK  (to  check  "Good  News")  in¬ 
terval,  every  400  milliseconds.  Section  4  provides  periodic 
entry  (every  100  milliseconds)  into  the  R TABLE  subroutine  at 
each  of  the  nodes.  Sections  3  and  4  use  portions  of  Section 
2's  code  in  order  to  reduce  redundancy  of  code.  Initiali¬ 
zation  of  all  the  algorithm's  tables,  arrays,  constants, 
and  flags  are  accomplished  by  Section  5.  When  topological 
changes  are  required,  Section  6  is  added.  The  code  of 
Sections  1  through  5  remains  the  same  throughout  all  the 
simulation  runs.  It  is  Section  6  that  gets  changed  whenever 
a  topological  change  is  desired.  A  more  detailed  discussion 
about  the  way  the  changes  are  accomplished  is  given  in  a 
later  section  of  this  chapter.  Section  7,  as  in  the  baseline 
program,  is  the  simulation  timimg  control  section.  As  ex¬ 
plained  in  Chapter  VI,  it  controls  the  traffic  volume  in¬ 
tensity  levels  and  statistical  output.  The  order  of  dis¬ 
cussion  for  the  sections  is  Section  5»  Section  2,  Section  3, 
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Section  4,  and  Section  1. 

As  stated  earlier.  Section  5  initializes  all  the  tables, 
arrays,  constants  and  flags  used  by  the  algorithm  as  a  whole 
by  activating  the  INITIALIZE,  INITIALIZE_D,  COMPUTE J),  and 
RTABLE  subroutines  for  each  of  the  nodes.  The  GENERATE 
instruction  generates  only  one  transaction  (packet).  The 
purpose  of  the  packet  is  to  move  through  the  various  HELP 
instructions  activating  the  subroutine  whose  name  is  the  A 
operand  of  the  instruction.  The  B  through  G  operands  (PI  to 
P6  are  the  values  of  the  parameters  1  to  6  of  the  packet 
but  are  used  only  for  dummy  arguments  because  of  the  ins¬ 
truction  format.  The  moves  through  the  HELP  instructions 
starts  at  those  given  for  Albany  and  terminates  at  the  TERM¬ 
INATE  instruction  after  activating  Tinker's  initializing 
subroutines.  The  packet  does  not  move  on  to  the  next  HELP 
instruction  until  the  present  subroutine  that  it  has  activated 
is  completed. 

Section  2  gives  the  code  necessary  to  enter  MESGEN 
under  the  FTICK  periodic  interval.  The  code  denoted  by  the 
encircled  "A"  generates  a  packet  every  103  milliseconds, 
assigns  priority  status  (parameter  2  »  2)  and  assigns  the 
special  status  packet  length  of  339  bits  (parameter  3*3) 
to  the  packet.  The  reason  for  generating  a  packet  every 
103  msec  instead  of  exactly  100  msec  is  to  help  ensure  that 
not  more  than  one  status  packet  is  being  processed  by  the 
same  section  of  code  at  the  same  time.  The,  packet 
enters  the  SPLIT  block  labeled  ALBME  where  a  duplicate  packet 

11? 


TT.^L'  »- 


■% 


with  the  same  parameter  values  is  generated.  One  packet 
goes  to  the  label  ANDME  (Andrews*  equivalent  MESGEN  code). 

The  first  instruction  at  each  of  the  other  node's  MESGEN 
code  is  a  split  instruction  which  sends  another  copy  of 
the  packet  to  the  next  succeeding  node's  MESGEN  code  for 
their  processing.  In  this  fashion,  the  MESGEN  section  of 
simulation  code  at  each  node  is  executed  almost  simul¬ 
taneously.  SAVEVALUE  122  is  set  to  a  value  of  one  (denoting 
"BAD  NEWS"  processing  in  Albany's  MESGEN  subroutine  ALMSGN). 
The  sequence  of  instructions  QUEUE,  SEIZE,  DEPART,  and  RE¬ 
LEASE  is  used  to  ensure  the  execution  of  the  code  denoted 
by  the  encircled  "B"  by  only  one  status  packet  at  a  time. 

The  packet  has  Albany  assigned  as  the  source  node  (parameter 
4=1)  and  SAVEVALUE  125  is  set  to  the  value  0.  The  HELPC 
ALMSGN  block  is  entered  next  to  activate  Albany's  MESGEN 
subroutine.  If  the  subroutine  has  in  fact  generated  a  status 
packet,  (SAVEVALUEs  57  -  64  contain  Albany's  status  packet 
information),  then  the  value  of  SAVEVALUE  125  is  changed  from 
its  previous  value  of  0  to  a  value  of  1.  If  ALMSGN  has 
changed  the  value  of  SAVEVALUE  125,  the  TEST  L  instruction 
sends  the  packet  to  the  instruction  labeled  ALBNY.  If  no 
status  packet  information  was  generated  by  ALMSGN,  the  pac¬ 
ket  terminates.  Whenever  the  MESGEN  determines  a  status 
packet  is  necessary,  SAVEVALUEs  are  used  to  store  the  actual 
information.  The  status  information  is  not  actually  assigned 
to  the  packet  which  activated  the  MESGEN  subroutine*  The  pac¬ 
kets  purpose  is  only  to  activate  the  appropriate  subroutine 
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to  manipulate  (read  or  modify)  the  contents  of  the  SAVE- 
VALUEs  which  represent  status  packet  information.  Assume 
status  packet  information  is  generated,  the  packet  enters 
the  instruction  labeled  ALBNY  where  a  duplicate  copy  is 
sent  to  the  instruction  labeled  ANDW1.  The  original  packet 
falls  through  to  the  ASSIGN  statement  where  parameter  1  is 
loaded  with  Albany's  ID  as  the  destination  node.  At  ALME1, 
the  sequence  of  instructions  QUEUE,  SEIZE,  DEPART,  and  RE¬ 
LEASE  is  used  to  ensure  the  execution  of  the  code  denoted 
by  the  encircled  "C"  and  is  executed  by  only  one  packet  at 
a  time.  Because  status  information  was  generated,  the  al¬ 
gorithm's  database  must  be  updated  to  reflect  the  new  current 
information.  The  HELPC  ALUPDT  instruction  activates  Albany's 
version  of  UPDATE  to  update  Albany's  LD  matrix  with  the  in¬ 
formation  contained  in  SAVEVALUEs  57  -  64.  The  HELPA  ALIN ID 
and  HELPA  ALCOMN  recomputes  Albany's  D  table  and  associated 
arrays.  At  ALME2,  a  similar  instruction  sequence  to  inhibit 
simultaneous  processing  of  packets  is  executed.  The  packet 
enters  the  HELPC  ALRTBL  instruction  which  activates  Al¬ 
bany's  version  of  the  RTABLE  subroutine.  Link  routing  arrays 
SROUTE  and  ROUTE  are  then  recalculated.  Having  accomplished 
its  intended  purpose  and  no  longer  needed  at  the  Albany  node, 
the  packet  enters  the  TERMINATE  block  denoted  by  the  encirc¬ 
led  ”D"  and  is  removed  from  the  system. 

Albany  has  now  processed  its  own  status  packet  infor¬ 
mation  but  the  other  nodes  also  need  to  process  Albany's 
status  information.  The  SPLIT  instruction  labeled  ALBME 
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had  previously  sent  a  packet  to  the  instruction  labeled 
ANDW1.  In  the  code  denoted  by  the  encircled  "E",  a  packet 
is  generated  for  each  of  the  other  nodes  with  the  appropriate 
destination  node's  ID  number  assigned  to  parameter  1  of  that 
packet.  Each  packet  is  transferred  to  be  input  processed 
as  any  other  packet  is  processed. 

As  indicated  earlier  in  this  section,  MESGEN  processing 
is  identical  for  each  of  the  nodes.  The  portion  of  code 
from  ALBME  down  to  and  including  the  encircled  "E"  is  dup¬ 
licated  for  each  node.  The  only  changes  in  code  are  the 
SAVEVALUE  instructions.  Each  node  has  its  own  set  of  SAVE- 
VALUEs.  A  complete  listing  of  the  SAVEVALUEs  and  those 
Assigned  to  each  node  is  given  in  Appendix  E. 

Section  3  enables  the  periodic  entry  into  MESGEN  for 
each  of  the  nodes  under  STICK  time  interval.  In  order  to 
reduce  the  chances  of  attempting  to  process  two  status  pac¬ 
kets  at  the  same  time,  packets  are  generated  every  401  msec 
instead  of  the  advertised  400  msec.  As  in  the  FTICK  code, 
the  packet  is  assigned  priority  status  (parameter  2  *  2) 
and  the  unique  packet  length  of  389  bits  (parameter  3  *  3)» 

At  ALBY  (for  Albany),  a  duplicate  packet  is  sent  to  ANDE 
(for  Andrews)  while  the  original  packet  falls  through  into 
the  SAVEVALUE  instruction.  SAVEVALUE  122  is  set  to  zero 
to  denote  "GOOD  NEWS"  processing  at  Albany  and  the  packet 
is  transferred  to  ALBM1  (in  the  previous  FTICK  coding). 

The  processing  of  the  packet  is  identical  to  that  of  the 
previous  packet  processing  discussion  only  the  SAVEVALUE  122 
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value  of  0  causes  the  MESGEN  code  to  check  for  new  opera¬ 
tional  links  and  the  absence  of  link  congestion  or  node 
saturation  (GOOD  NEWS).  The  rest  of  the  code  in  Section  3 
sets  the  appropriate  SAVEVALUEs  to  zero  for  each  of  the  res¬ 
pective  nodes  then  transfers  the  packets  to  the  proper  entry 
point  in  each  node's  FTICK  coding. 

Periodic  entry  into  the  RTABLE  subroutines  at  the  various 
nodes  is  accomplished  in  Section  4.  Packets  are  generated 
every  100  milliseconds  and  since  these  packets  are  not  trans¬ 
mitted  no  values  are  assigned  to  any  of  their  parameters. 

The  sole  purpose  of  a  packet  generated  in  this  section  is 
to  cause  the  activation  of  a  node's  RTABLE  subroutine.  After 
generation,  a  packet  enters  the  SPLIT  instruction  labeled 
ALBN  (for  Albany)  whereupon  a  duplicate  copy  is  sent  to  ANDS 
(for  Andrews).  The  original  packet  falls  through  to  the 
TRANSFER  instruction  and  is  transferred  to  ALME2  (in  the  pre¬ 
vious  FTICK  code).  The  rest  of  Section  4's  code  creates  and 
transfers  packets  to  the  respective  entry  points  in  that 
node's  code  to  activate  the  appropriate  RTABLE  subroutine. 

Now  that  the  activation  of  the  algorithm's  Background 
subroutines  have  been  discussed,  changes  to  the  main  simu¬ 
lation  program,  Section  1,  will  be  explained.  Since  each 
node  requires  the  same  general  changes,  the  changes  necessary 
for  Albany's  code  are  explained.  The  first  required  change 
comes  as  a  requirement  to  activate  Albany's  PAKROUTE  sub¬ 
routine.  The  encircled  "F"  shows  the  code  necessary  to 
activate  and  use  the  results  of  the  subroutine  PI  through  ?6 
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(parameters  1  through  6  of  the  packet),  which  contain  all 
the  necessary  information  needed  by  the  subroutine  to  se¬ 
lect  the  trunk  over  which  to  route  the  packet.  When  the  pac¬ 
ket  enters  the  HELPC  ALPKRT  instruction,  the  subroutine  uses 
the  values  of  PI  (destination  node  ID)  and  P4  (source  node 
ID)  in  the  calculation  to  select  the  best  trunk.  The  selec¬ 
ted  trunk’s  ID  number  is  stored  in  the  FULLWORD  SAVEVALUE  1. 
When  control  is  returned  to  the  GPSS  simulation  program,  the 
packet  enters  the  following  ASSIGN  block.  The  trunk  ID  num¬ 
ber  in  FULLWORD  SAVEVALUE  1  is  assigned  to  packet  parameter  5* 
The  TRANSFER  block  uses  the  value  of  parameter  5  and  FUNCTION 
R0UT1  to  transfer  the  packet  to  the  correct  trunk  queue. 

Changes  are  also  required  to  the  R0UT1  function,  encir¬ 
cled  "G".  Packets  can  be  sent  to  any  one  of  Albany's  seven 
trunks,  terminated  locally  or  be  removed  from  the  system  be¬ 
cause  the  packet  is  undeliverable  (the  destination  node  of 
the  packet  is  nor, -operational) .  Thus  nine  entries  are  re¬ 
quired  in  the  function.  PAKROUTE  returns  one  of  nine  numbers 
(1  -  9)  to  be  placed  in  parameter  5  of  the  packet.  If  the 
packet  value  is  1  through  7»  the  packet  is  sent  to  the  indi¬ 
cated  trunk  for  transmission.  If  the  value  is  an  8,  the  pac¬ 
ket  is  sent  to  TERM1  for  local  termination  processing,  and 
if  the  value  is  a  9»  the  packet  is  undeliverable  and  is  sent 
to  a  terminate  block  for  removal  from  the  system.  Local 
termination  processing,  indicated  by  the  encircled  "H",  in¬ 
cludes  changes  to  the  program  to  check  and  see  if  the  packet 
represents  a  status  packet.  The  packet's  length  is  checked 
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to  see  if  the  length  corresponds  to  the  length  of  a  status 
packet.  If  it  does,  the  packet  is  transf erred  to  ALME1  where  , 
the  packet  causes  activation  of  Albany's  UPDATE  subroutine. 
Inside  of  Albany's  UPDATE  subroutine,  ALUPDT,  the  correct 
set  of  SAVEVALUEs,  containing  the  new  status  information  for 
the  node  whose  status  packet  is  being  processed,  is  accom¬ 
plished  by  a  calculation  using  the  packet's  source  node  ID 
number.  After  UPDATE,  the  same  packet  causes  execution  of 
INITIAL IZE_D ,  C0MPUTE_D,  and  R TABLE  to  recalculate  and  up¬ 
date  the  algorithm's  database  then  terminates. 

Since  status  packets  are  being  transmitted  through  the 
network,  changes  are  required  to  the  portion  of  code  repre¬ 
senting  the  transmission  of  packets.  For  example,  a  status 
packet  from  Albany  is  to  be  transmitted  to  Ft.  Detrick  (de¬ 
noted  by  the  encircled  "I").  The  packet  enters  the  queue 
ALQ22,  seizes  the  trunk  ALTK3,  and  departs  the  queue.  To 
determine  how  much  transmission  time  is  required  for  the 
packet,  parameter  3  (containing  the  length  of  the  packet) 
and  FUNCTION  ALF22  (denoted  by  the  encircled  "J")  is  used. 
Since  the  value  of  parameter  3  is  a  3*  the  packet  is  sent 
to  ALC22  where  the  packet  accumulates  16  msec  of  trans¬ 
mission  time.  After  acquiring  the  specified  amount  of 
time,  the  packet  releases  the  trunk,  ALTK3.  and  is  trans¬ 
ferred  to  the  second  SCM  processor  at  Ft.  Detrick  (FTDE2) 
for  further  processing. 
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Procedures  for  Running  the  Modified  Program 


The  first  set  of  four  runs  is  with  the  network  totally 
operational,  that  is,  all  trunks,  links,  and  nodes  up  and 
functioning.  The  same  method  of  running  the  baseline  pro¬ 
gram  with  one  run  per  traffic  intensity  spike  of  60%,  ?0%, 

80%,  and  90%  of  network  saturation  is  used  for  running  the 
algorithm  version  of  the  simulation  program.  Appendix  F 
shows  the  graphical  and  tabular  results  of  this  first  set  of 
simulation  runs. 

The  program  code  used  to  simulate  the  network  losses  is 
the  portion  of  the  main  program  code  with  the  comment  title, 
"CODE  FOR  ENTRY  INTO  CONNECTIVITY  CHANGE"  (Section  6),  and 
the  algorithm's  CONN_CHANGE  subroutine.  A  discussion  is 
given  detailing  the  program  changes  required  for  each  of  the 
different  types  of  network  losses.  For  ease  of  explanation, 
one  specific  example  is  discussed  in  each  of  the  three  types 
of  topological  changes. 

Trunk  Loss  Simulation.  Figure  26  shows  the  code  of  the 
main  program  necessary  to  drop  the  third  trunk  in  the  Albany  - 
Ft.  Detrick  link.  There  are  two  parts  of  code  in  Figure  26. 
Part  1  generates  one  packet  25  seconds  into  the  simulation.  The 
packet  is  assigned  priority  status  and  the  length  of  a  status 
packet.  An  identical  packet,  including  the  same  parameter 
values,  is  created  in  the  SPLIT  block  (denoted  by  the  encir¬ 
cled  "A")  and  sent  to  label  SECPR.  The  original  packet  falls 
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through  into  the  SAVEVALUE  instructions  where  SAVEVALUEs  121, 
126,  and  12?  are  loaded  with  values  used  in  ALCCHG,  Albany's 
CONN_CHANGE  subroutine  (Figure  27).  Zero  is  loaded  into 
SAVEVALUEs  121  and  126  to  set  flags  to  denote  a  change  in 
trunk  status  and  that  a  trunk  is  to  be  dropped  respectively. 
One  in  SAVEVALUE  122  sets  the  flag  to  denote  "BAD  NEWS"  pro¬ 
cessing  is  to  occur  in  Albany's  version  of  MESGEN  after  the 
return  from  ALCCHG.  A  two  in  SAVEVALUE  127  indicates  that 
a  trunk  will  be  dropped  in  Albany's  link  2  to  Ft.  De trick. 

The  HELPC  ALCCHG  instruction  causes  control  to  be  passed  to 
the  ALCCHG  subroutine.  In  the  subroutine,  control  is  passed 
from  statement  4  to  statement  100.  The  ALLINS  array  is  used 
to  store  the  number  of  trunks  in  a  particular  link  indexed 
by  that  link  number.  Since  SAVEVALUE  126  has  a  0  in  it, 
control  is  passed  to  the  next  statement  where  a  2  is  stored 
into  ALLINS  (2).  Originally  there  were  three  trunks  in  link 
2  to  Ft.  Detrick,  but  now  there  are  two,  in  effect  dropping 
a  trunk  from  the  link.  Control  is  returned  to  the  GPSS 
simulation  program  and  the  packet  is  transferred  to  a  portion 
of  code  in  Section  2.  MESGEN  is  entered  to  see  if  dropping 
of  the  trunk  requires  the  generation  of  new  status  infor¬ 
mation.  If  not,  the  packet  is  removed  from  the  system. 
Otherwise  its  further  processing  is  as  described  earlier  in 
reference  to  the  processing  of  a  status  packet.  Similar  pro¬ 
cessing  goes  on  starting  at  label  SECPR,  encircled  "B", 

Figure  26,  to  simulate  Ft.  Detrick's  dropping  of  a  trunk  in 
its  link  to  Albany. 
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In  order  to  bring  the  trunks  back  up  to  operational 
status  at  Albany  and  Ft.  De trick,  the  code  in  Part  2, 

Figure  26,  is  exercized.  Forty-six  seconds  into  the 
simulation,  a  packet  is  generated  (the  trunks  have  been 
down  for  twenty-one  seconds).  The  packet  is  assigned 
priority  status,  given  status  packet  length,  and  a  dupli¬ 
cate  copy  created  and  sent  to  label  SECP1.  The  only  change 
in  the  SAVEVALUEs  is  to  SAVEVALUE  122  and  126.  SAVEVALUE  122 
is  loaded  with  a  zero  to  denote  "GOOD  NEWS"  processing  is  to 
occur  when  MESGEN  is  entered  in  later  code.  SAVEVALUE  126 
is  loaded  with  a  one  to  flag,  in  the  ALCCHG  subroutine,  the 
reestablishment  of  the  dropped  trunk  back  into  the  link 
between  Albany  and  Ft.  Detrick.  SAVEVALUEs  retain  their 
values  throughout  and  the  simulation  unless  specifically 
changed,  hence  no  need  to  change  SAVEVALUE  12?.  When  the 
ALCCHG  subroutine  is  activated  by  the  packet  entering  the 
HELPC  ALCCHG  instruction,  the  values  of  ALLINS  (2)  is 
changed  back  to  a  three,  essentially  bringing  the  trunk  back 
up  to  operational  status.  When  control  returns  to  the  GPSS 
simulation  the  packet  transfers  to  the  same  portion  of  code 
in  Section  2  and  is  processed  similarly  as  described  earlier. 

The  duplicate  copy  of  the  packet  sent  to  SECP1,  encir¬ 
cled  "C",  Figure  26,  accomplishes  for  Ft.  De trick  the  same 
job  as  the  packet  did  for  Albany.  The  trunk  at  Ft.  Detrick 
returns  as  one  of  the  three  operational  trunks  in  the  link 
to  Albany  and  Ft.  De trick’s  version  of  MESGEN  is  entered  to 
determine  if  new  status  information  should  be  generated. 


Link  Loss  Simulation.  In  this  set  of  examples,  program 
changes  to  Section  6  of  the  main  program  are  made  to  faci¬ 
litate  simulating  the  loss  of  a  link  between  two  nodes. 

Again,  for  use  of  explanation,  the  specific  example  of  losing 
the  link  connecting  Gentile  and  Ft.  Detrick  is  used.  Fig¬ 
ure  28  shows  the  code  necessary  to  simulate  the  condition. 

As  in  the  trunk  example,  there  are  two  pants  to  Section  6. 
Part  1  is  the  code  used  in  simulating  the  loss  of  the  link, 
and  Part  2  is  the  code  used  to  bring  the  link  back  up.  The 
packet  is  processed  in  similar  fashion  as  described  in  the 
trunk  loss  example  above  with  exceptions  in  values  stored 
in  SAVEVALUEs.  Gentile’s  version  of  CONN_CHANGE  (GECCHG) , 
uses  the  values  in  SAVEVALUEs  150,151,  155*  and  I56.  A  one 
in  SAVEVALUE  150  flags  the  section  of  code  in  GECCHG  neces¬ 
sary  to  provide  a  change  in  link  status.  A  one  in  SAVEVALUE 
151  flags  the  future  execution  of  MESGEN  to  check  for  "BAD 
NEWS" .  A  zero  in  SAVEVALUE  155  flags  the  execution  of  code 
in  GECCHG  to  drop  the  link  whose  value  is  in  SAVEVALUE  156 
(1  «*>  Gentile’s  first  link).  When  GECCHG  (Figure  29),  is 
activated,  statement  4  passes  control  to  statement  5»  In¬ 
dexes  into  Gentile's  LD  table  (GEN1  and  GEN2),  sure  set  up 
with  GEN1  taking  on  Gentile's  ID  number  and  GEN2  taking  on 
Ft.  Detrick’s  ID  number.  At  the  encircled  "A"  of  Figure  29, 
execution  of  the  instruction  determines  whether  the  link 
change  involves  dropping  the  link  from  or  adding  it  back 
to  the  Gentile  node.  If  the  link  is  to  be  dropped,  (SAVE¬ 
VALUE  155  s  0),  control  is  passed  to  statement  10,  other- 
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Fig  29.  CONN_CHANGS  Subroutine  for  Gentile 


wise  control  is  passed  to  statement  15 •  At  statement  10 
GESTAT  is  set  to  a  -1  (used  shortly  to  set  the  appropriate 

entry  in  Gentile's  LD  matrix  and  the  link  number  in  Gentile's 
NC  array  to  negative  values).  XXBUP  (GEBOP  for  Gentile)  is 
an  array  denoting  the  current  status  of  the  Gentile  links. 

A  one  in  the  array  indicates  the  link  is  up  and  a  1000 
indicates  the  link  is  down.  Since  SAVEVALUE  156' s  value  is 
a  1,  GEBUP(l)  is  set  to  1000  to  indicate  the  loss  of  the  link. 
(The  GEBUP  array  is  used  in  MESGEN  to  compare  it  against  the 
previous  status  of  the  links  contained  in  the  XXSTUS  array 
(GESTUS  for  Gentile  )).  Control  is  then  passed  to  statement 
20  where  the  value  of  the  LD  matrix  indexed  by  GEN1  and  GEN 2 
is  made  negative  (used  later  in  GEINID  and  GEC0MD  to  recalcu¬ 
late  the  D  matrix  for  Gentile).  Index  I  value  is  calculated 
to  equal  the  value  of  GEN2  and  the  corresponding  value  of 
Gentile's  array  is  made  negative.  Control  is  then  returned 
to  the  GPSS  simulation  program.  In  order  to  compensate  for 
this  new  condition,  the  packet  is  transferred  to  a  portion 
of  code  in  Section  2  (GENM1)  used  to  handle  the  processing 
of  Gentile's  version  of  MESGEN  and  Gentile's  association 
entries  into  UPDATE,  INITIALED,  C0MPUTE_D,  and  R TABLE.  The 
duplicate  copy  of  the  packet  sent  to  SECPR  (encircled  "A"  on 
Figure  28),  receives  similar  processing  for  the  Ft.  Detrick 
node  as  discussed  above  for  the  Gentile  node. 

After  forty-six  seconds  of  simulation  elapses,  the  link 
between  Gentile  and  Ft.  Detrick  is  brought  back  up  into  op¬ 
erational  status  by  the  execution  of  the  code  in  Part  2  of 


Figure  28.  SAVEVALUEs  are  changed  to  flag  the  execution  of 
code  in  GECCHG  and  DECCHG  to  reverse  their  previous  actions 
and  reestablish  the  link  between  Gentile  and  Ft.  De trick. 
CONN_CHANGE  is  entered  to  bring  the  link  back  up,  MESGEN 
builds  new  status  information  which  is  sent  locally  and  to 
the  other  nodes.  UPDATE  makes  the  appropriate  changes  to 
the  LD  matrix,  INITIALIZE_D  reinitializes  the  D  matrix  and 
associated  arrays,  COMPUTE_D  recomputes  the  new  D  matrix  and 
R TABLE  recomputes  the  link  routing  arrays  SROUTE  and  ROUTE. 
The  processing  necessary  to  drop  a  link  versus  what  is  neces¬ 
sary  to  drop  a  trunk  is  more  complicated  because  a  major 
change  in  network  connectivity  is  occurring.  New  status 
information  will  be  generated  when  MESGEN  is  entered  later 
or  thus  everything  must  be  set  up  for  other  subroutines  to 
simulate  the  condition  correctly. 

Node  Loss  Simulation.  The  code  necessary  to  accomplish 
the  removal  of  a  node  from  the  network  is  more  complicated 
than  that  of  either  of  the  two  previous  examples.  Each  of 
the  nodes  connected  to  the  non-operational  node  must  remove 
their  respective  link  to  the  down  node  from  operational 
status  and  throw  away  the  packets  whose  destination  is  the 
downed  node.  In  this  simulation  program,  even  though  the 
last  node  is  disconnected  from  the  rest  of  the  network  it 
8 till  generates  packets  and  processes  them  as  if  it  were  op¬ 
erational.  It  was  easier  to  incorporate  this  change  than 
remove  the  node*s  code  completely  when  it  goes  down  and  re¬ 
place  the  code  when  the  node  comes  back  up  again.  When  the 
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node*s  PAKROUTE  subroutine  is  activated,  instead  of  putting 
the  appropriate  trunk  number  into  the  related  FULLWORD  SAVE- 
VALUE  an  integer  number  is  placed  in  the  related  FULLWORD 
SAVEVALUE  which,  when  control  is  returned  to  the  main  simu¬ 
lation  program,  causes  the  packet  to  be  sent  to  a  TERMINATE 
block  for  removal  from  the  system. 

Similar  changes  were  required  in  each  of  the  other  node's 
PAKROUTE  subroutines.  If  the  packet's  destination  is  the 
downed  node,  the  appropriate  integer  number  is  placed  into 
the  related  FULLWORD  SAVEVALUE  which  will  result  in  the  pac¬ 
kets  eventual  removal  from  the  system.  Figure  30  shows  Sec¬ 
tion  6's  code  required  to  simulate  the  loss  and  reestablish¬ 
ment  of  the  Ft.  Detrick  node.  As  in  the  previous  two  examples 
there  are  two  parts  to  the  code.  The  nodes  connected  to 
Ft.  Detrick  (Albany,  Andrews,  Hancock,  Gentile,  and  Tinker) 
each  have  their  own  section  of  code  in  Part  1  to  simulate 
the  loss  of  Ft.  De trick.  In  each  node's  code,  their  respec¬ 
tive  SAVEVALUEs  are  set  up  to  drop  their  link  to  Ft.  De trick 
and  CONN_CHANGE  is  executed  at  each  node.  Ft.  De trick's  code 
(starting  at  the  label  SECPD)  loads  its  SAVEVALUE  with  special 
flags.  SAVEVALUE  141  is  loaded  with  the  normal  value  of  1 
to  denote  "BAD  NEWS"  processing  in  MESGEN.  The  other  SAVE¬ 
VALUEs  are  loaded  with  dummy  values  which,  in  effect,  causes 
no  change  in  the  status  of  any  of  its  links  in  DECCHG.  The 
key  SAVEVALUE  is  146.  It  is  loaded  with  a  value  of  1.  Its 
purpose  will  be  explained  shortly. 

When  CONN  CHANGE  has  been  executed  at  each  connected 
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node  and  at  Ft.  De trick,  the  individual  status  are  trans¬ 
ferred  to  portions  of  code  in  Section  2  to  activate  the 
other  subroutines  necessary  to  complete  the  generation  of 
new  status  information,  update  and  recalculate  the  node's 
database  and  to  broadcast  each  node's  new  status  information 
to  the  rest  of  the  network.  After  46  seconds  of  simulation 
time,  the  GENERATE  block  in  Part  2,  generates  i  packet.  The 
packet  and  its  copies  (created  from  the  SPLIT  blocks)  process 
through  their  respective  node's  reseting  the  SAVEVALUEs  which 
are  needed  to  simulate  Ft.  De  trick  coming  back  up.  CONN__CHANGE 
is  executed  at  each  node  except  Ft.  De trick.  Due  to  the  dummy 
values  in  Ft.  De trick's  SAVEVALUEs,  if  DECCHG  were  executed, 

Ft.  Detrick's  link  to  Albany  would  be  simulated  as  being  down. 

After  return  from  CONN_CHANGE,  the  various  status  packets 
are  sent  to  portions  of  code  in  Section  2  for  further  proces¬ 
sing  necessary  to  recalculate  and  update  their  respective 
node's  database. 

The  only  program  changes  to  the  algorithm  were  to  each 
node '8  version  of  PAKROUTE.  The  same  four  changes  were  re¬ 
quired  at  every  node's  PAKROUTE  subroutine  except  Ft.  De trick's 
which  required  three  changes.  Figure  31  shows  the  resulting 
PAKROUTE  subroutine  for  Albany.  The  encircled  MA"  shows  the 
four  instruction  changes.  The  first  instruction  tests  to 
see  if  Ft.  Detrick  is  down  (SAVEVALUE  146  =1).  If  the  node 
is  not  down,  control  is  passed  to  statement  3  and  normal  pro¬ 
cessing  occurs.  If  Ft.  De trick  is  down,  the  next  instruction 
tests  to  see  if  the  destination  of  this  packet  is  Ft.  Detrick. 

* 

I 

i  ( 
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PA<KO«TE  LOP<S  AT  THE  CUEJE  LENGTHS  OF  THE  INOTVIOUAL  TKtmS 
MAKING  (Ip  THE  LIN<  4)  SELfS’tO  HY  SivOUTt  OR  R  JUl  t  (SOURCE 
NOO:  CR  TANJtM  N03E)  «N 0  iit-tCTS  THE  TRUNR  WITH  THE  L?AS 


FSA(l)  =  a 

Fig  31.  PAKROUTE  for  Albany 
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Fig  31.  (CONT*D) 


(packet  parameter  1=3)*  If  the  destination  is  not 
Ft.  Detrick,  control  is  passed  to  statement  3  for  normal 
processing.  If  Ft.  Detrick  is  the  destination,  FULLWORD 
SAVEVALUE  1  is  set  to  9»  Control  is  then  passed  to  state¬ 
ment  1000  for  return  to  the  GPSS  program.  Back  in  the 
simulation  program  the  value  in  FULLWORD  SAVEVALUE  1  is  moved 
into  packet  parameter  5  and  the  packet  is  transferred  to 
UNDE1  for  removal  from  the  network. 

Figure  32  shows  the  resultant  code  for  Ft.  Detrick.  The 
code  shown  by  the  encircled  "A"  are  the  three  changes  added 
to  the  PAKROUTE  subroutine  at  Ft.  Detrick.  If  the  node  is 
not  down,  control  is  passed  to  statement  3  for  normal  pro¬ 
cessing.  If  the  node  is  down,  it  can't  transmit  any  packets, 
thus  13  is  stored  in  FULLWORD  SAVEVALUE  3.  Control  is  passed 
to  statement  1000  for  return  to  the  main  simulation  program. 
Upon  return,  the  value  in  FULLWORD  SAVEVALUE  3  is  moved  in 
packet  parameter  5«  The  value  of  FULLWORD  SAVEVALUE  3  (1^) 
is  used  by  FUNCTION  R0UT3  to  send  the  packet  to  UNDE3  for 
termination  and  removal  from  the  system. 

The  following  chapter  contains  an  analysis  of  the  algo¬ 
rithm's  ability  to  adapt  to  traffic  fluctuations  while  net¬ 
work  topology  changes  are  occurring.  The  chapter  also  con¬ 
tains  a  comparison  summary  of  the  packet  delay  results  ob¬ 
tained  by  the  simulation  exercise  and  the  packet  delay  re¬ 
quirements  as  set  forth  in  DCA's  Performance  Specifications 
for  the  AUTODIN  II  network.  The  summary  includes  specific 
examples  of  topological  changes  and  traffic  fluctuation 


Fig  32.  PAKROUTE  for  Ft.  De trick 
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levels  not  satisfying  the  Performance  Specifications  based 
on  statistical  Hypothesis  tests  and  confidence  intervals, 
and  a  short  section  on  the  disturbance  periods  (the  time 
between  the  traffic  spike  and  the  return  of  the  network 
to  the  system  mean  delay).  The  paper  concludes  with  rec- 
commendations  on  possible  network  reconfiguration  changes 
and  ideas  on  further  simulation  exercises. 


VIII  Simulation  Results  and  Analysis 


It  should  first  be  stated  that  the  development  of  the 
simulation  model,  and  thus  its  packet  delay  results  evolved 
from  the  author* s  interpretation  and  implementation  of  DCA's 
preliminary  design  requirements  package  which  included  traf¬ 
fic  volume  figures,  internodal  packet  exchanges,  and  the 
network  Base  Routing  matrix.  Since  the  actual  algorithm 
code  was  not  available,  the  author's  implementation  of  the 
algorithm  in  Fortran  is  also  susceptible  to  interpretation. 

It  is  the  author's  opinion  that  given  the  model  used  and  the 
methodology  of  its  development,  the  conclusions  (in  Chap¬ 
ter  IX),  statements,  and  analysis  of  the  delay  results  in 
this  report  are  factual. 

Algorithm  Adaptability 

Webster's  New  Collegiate  Dictionary  defines  adaptation 
as  "the  act  or  process  taken  to  adjust  to  environmental 
conditions".  The  ability  to  adapt  to  these  changing  en¬ 
vironmental  conditions  (traffic  volume  fluctuations  and 
network  topology)  is  a  measure  of  the  hierarchical 
routing  algorithm's  performance.  If  the  algorithm  causes 
undue  excessive  packet  delay  in  the  process  of  adapting  to 
and  compensating  for  changing  environmental  conditions,  it 
is  said  to  be  not  well-behaved.  Without  any  precedence  to 
go  by  in  defining  "undue,  excessive  packet  delay",  the 
author  appends  the  following  values  to  the  definition!  for 
any  mean  packet  delay  from  the  simulation  equal  to  or  greater 
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than  570  msec  for  short  packets  and  795  msec  for  long  pac¬ 
kets,  the  algorithm  is  said  to  be  not  well-behaved.  Another 
measure  would  be  2025  msec  for  the  maximum  packet  delay  of 
short  packets  and  2250  msec  for  the  maximum  packet  delay  of 
long  packets.  These  values  (defined  by  the  author)  are  one 
and  one  half  times  the  packet  delay  requirements  given  in 
Figure  25,  Chapter  VI. 

The  author  developed  an  additional  performance  criterion 
as  a  measure  to  determine  another  behavioral  pattern  of  the 
algorithm.  The  "disturbance  period"  is  defined  to  be  the 
time  it  takes  for  the  network  to  return  to  within  a  95^ 
Confidence  Interval  of  the  system  mean  packet  delay  after 
the  occurrence  of  a  traffic  spike.  The  system  mean  packet 
delay  is  the  delay  value  in  parenthesis  at  the  30  second  time 
frame  on  each  graph.  Also  for  identification  purposes,  the 
delay  values  in  parentheses  during  disturbance  periods,  in 
graphs  of  Appendices  F  through  I,  are  the  maximum  packet 
delays  incurred  from  packets  terminating  (locally  delivered) 
within  the  previous  one  second  time  interval. 

Having  defined  two  performance  characteristics  to  de¬ 
termine  the  adaptability  of  the  routing  algorithm,  four 
sample  cases  are  discussed  with  respect  to  the  algorithm's 
ability  to  adapt  to  the  changing  environmental  conditions. 

Selected  Results  Explained 

The  four  cases  being  discussed  are  (1)  the  fully  op- 
(  erational  network  with  the  algorithm  versus  the  baseline 
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program,  (2)  an  example  from  the  trunk  loss  program  runs, 

(3)  an  example  from  the  link  loss  program  runs,  and  (4) 
an  example  from  the  node  loss  program  runs. 

Fully  Operational  Network  Versus  Baseline.  In  this 
section,  the  graphs  of  the  baseline  results  (Figures  22, 

23  and  24  of  Chapter  VI)  are  compared  against  the  graphs 
of  the  fully  operational  network  in  Appendix  F  (Figures 
F-l,  F-2,  and  F-3). 

In  comparing  the  graphs  of  the  system  delay  (all  pac¬ 
kets),  Figures  22  and  F-l,  the  baseline  program  takes  nine 
seconds  to  return  to  equilibrium  whereas  the  algorithm  ver¬ 
sion  takes  only  five  seconds.  These  disturbance  periods 
mean  the  algorithm  can  react  to  traffic  fluctuations  in  in¬ 
tensity  faster  than  the  static  routing  methodology  of  the 
baseline  program.  Also  shown  in  these  graphs  is  the  maximum 
delay  times  experienced  by  any  packet  during  the  disturbance 
period.  The  maximum  delay  a  packet  experiences  in  the  al¬ 
gorithm  version  (I850  msec)  occurs  two  seconds  after  the  90$ 
spike.  In  the  baseline  program,  the  maximum  delay  (2830  msec) 
occurs  three  seconds  after  the  90$  spike.  The  mean  delays 
are  higher  in  the  algorithm  version  than  in  the  baseline 
but  this  is  due  to  additional  load  on  the  system  because  of 
the  system  status  packets  being  generated  and  broadcast 
throughout  the  network.  With  the  addition  of  the  algorithm, 
the  additional  status  packets  add  6$  to  the  traffic  volume 
in  the  network  over  and  above  what  the  baseline  was  processing. 
Also  as  a  matter  of  interest,  is  the  fact  that  all  the  packets 
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in  this  additional  6#  are  priority  packets  which  in  effect 
delay  the  vast  majority  of  the  other  packets  in  the  network 
which  are  non-priority.  But  even  with  this  additional  traf¬ 
fic  load,  the  algorithm  minimised  the  affect  of  the  traffic 
spike,  lowered  maximum  delay  times,  and  reacted  to  the  traf¬ 
fic  fluctuation  better  than  the  baseline  program. 

Figures  23  and  F-2  are  related  graphs  showing  the  system 
delay  for  non-priority  -  short  packets.  The  same  conclusions 
on  the  algorithm  performance  can  be  drawn  if  the  two  graphs 
are  compared  to  each  other.  The  disturbance  period  is  short¬ 
er  with  the  algorithm  (5  seconds  versus  7  seconds),  maximum 
delay  is  lower  (1850  msec  versus  2710  msec),  and  the  mean 
delays  during  the  disturbance  period  is  lower  for  the  baseline 
An  interesting  characteristic  of  the  algorithm  is  the  apparent 
immediate  reaction  to  the  traffic  spike.  One  second  after 
the  spike,  the  maximum  mean  delay  is  reached,  but  in  the  base¬ 
line  the  maximum  mean  delay  occurs  two  seconds  after  the  spike 

Figures  24  and  F-3  show  the  results  of  the  two  simula¬ 
tion  versions  for  the  system  delay  of  non-priority  -  long 
packets.  The  affect  of  the  algorithm  on  the  disturbance  per¬ 
iod  is  pronounced  in  comparing  the  delay  trends  at  the  90# 
saturation  level  (4  seconds  in  the  algorithm  version  and  9 
seconds  in  the  baseline). 

The  data  obtained  for  the  short  and  long  priority  pac¬ 
kets  cannot  really  be  compared.  As  stated  in  Chapter  VI, 
there  are  not  enough  user  priority  packets  generated  for 
qualitative  or  quantitative  analysis.  The  data  does  show 


though,  (in  Tables  IX  and  X  of  Chapter  VI,  and  Tables  F-l 
and  F-II)  that  the  delays  experienced  by  the  priority  pac¬ 
kets  did  not  exceed  the  mean  or  maximum  delay  constraints 
as  specified  in  the  DCA  Performance  Specifications  (Figure 
25,  Chapter  VI). 

Simulated  Tocology  Degradations.  One  of  the  character¬ 
istics  of  an  efficient  routing  algorithm  is  its  ability  to 
recognize  and  respond  in  a  prompt,  orderly  manner  to  net¬ 
work  topology  changes.  These  changes  could  consist  of  losses 
of  trunks,  complete  links,  or  even  nodes  in  the  network.  In 
this  section  of  the  chapter,  the  simulation  of  topological 
changes  are  made  selectively,  choosing  one  of  the  network's 
trunks,  links,  or  nodes,  and  dropping  it  from  the  network. 
While  the  facility  is  non-opera tional,  a  traffic  spike  occurs, 
requiring  the  algorithm  not  only  to  compensate  for  the  lost 
facility,  but  also  to  contend  with  the  spiked  increase  in 
traffic  volume.  The  selection  of  which  trunks,  links,  and 
nodes  to  drop  is  made  based  on  dropping  a  trunk  in  the  most 
heavily  utilized  link,  dropping  that  link  itself,  and  dropping 
the  most  vital  tandem  nodes  of  the  network.  In  order  to  have 
something  to  measure  in  the  disturbance  periods  of  the  top¬ 
ological  change  simulation  runs,  the  disturbance  periods 
of  the  fully  connected  network  simulation  runs  (Figures  F-l 
through  F-3)  are  used  as  control  samples.  For  the  graphs 
showing  packet  delays  for  all  packets  (Figure  F-l),  the 
disturbance  period  is  5  seconds  (from  time  frame  31  to  time 
frame  36) •  For  the  graphs  showing  packet  delays  for  short 


packets  (Figure  F-2),  the  disturbance  period  is  also  five 
seconds.  For  the  graphs  showing  packet  delays  for  long  pac¬ 
kets  (Figure  F-3)*  the  disturbance  period  is  four  seconds 
(from  time  frame  31  to  time  frame  35)* 

Sample  Trunk  Loss.  The  trunks  dropped  are  the 
trunks  in  the  links  connecting  Albany  -  Ft.  Detrick,  Ft.  De¬ 
trick  -  Gentile,  Andrews  -  Tinker,  Ft.  De trick  -  Tinker,  and 
Gentile  -  Tinker.  Each  of  the  five  examples  involves  running 
the  program  four  times,  once  per  saturation  spike.  In  the 
AUTODIN  II  network,  trunks  are  full  duplex  (two-way)  trans¬ 
mission  lines,  that  is,  traffic  is  both  sent  and  received 
on  the  same  trunk.  However,  in  the  simulation,  two  one-way 
trunks  are  required  to  effect  the  simulation  of  one  full  du¬ 
plex  trunk  in  the  proposed  network. 

Results  from  simulation  runs  involving  the  losses  of 
trunks  are  given  in  Appendix  G.  Each  example  of  the  gra¬ 
phical  and  tabular  results  is  preceded  by  a  page  informing 
the  reader  which  set  of  trunks  is  dropped  for  that  example. 

For  the  trunk  loss  example,  the  simulation  runs  simu¬ 
lating  the  loss  of  a  trunk  in  the  link  between  Gentile  and 
Tinker  (Figures  G-13  through  G-15)»  is  selected.  Although 
it  is  one  of  the  worst-case  examples  for  the  set  of  trunk 
loss  runs,  the  use  of  it  in  the  ensuing  discussion  merits  in¬ 
vestigation.  Even  with  the  loss  of  the  trunk,  the  disturbance 
periods  for  both  the  all-packet  graph  (Figure  G-13),  and  the 
short  packet  graph  (Figure  G-14),  equals  the  matching 
graphs  of  the  control  sample.  The  disturbance  period  for 

151 


the  long  packet  graph  (Figure  G-15)  is  5  seconds  which  is 
only  one  second  longer  than  that  of  the  long  packet  control 
sample.  Typically*  this  means  even  with  the  additional  bur¬ 
den  of  a  lost  primary  trunk,  the  algorithm  enables  the  net¬ 
work  to  withstand  a  large  traffic  spike  (90#  saturation 
worst-case)  and  still  return  to  within  a  95%  Confidence 
Interval  of  the  mean  system  delay  within  five  seconds  of  the 
time  of  the  spike.  If  another  traffic  spike  occurs  within 
five  seconds  of  the  first  spike,  the  network  would  undergo 
additional  stress  and  could  quite  possibly  become  not  well- 
behaved.  Since  the  network  had  already  attained  mean  delay 
equilibrium  by  the  time  the  trunk  came  back  up,  its  addition 
to  the  network  did  very  little  to  lower  the  system  mean  pac¬ 
ket  delay.  An  interesting  aspect  of  the  delay  results  during 
the  disturbance  period  shows  that  the  mean  and  maximum  packet 
delays  for  the  60,  70,  and  80  percent  saturation  runs  (Figures 
G-13  through  G-15)  are  less  than  those  of  the  control  sample 
(Figures  F-l  through  F-3).  This  could  imply  that  the  algo¬ 
rithm  calculates  better  paths  for  packets  to  take  while  un¬ 
der  moderate  stress.  Although  the  algorithm  is  well-behaved 
in  this  trunk  loss  sample,  the  mean  and  maximum  packet  delays 
for  90#  exceeds  the  DCA  delay  requirements. 

Sample  Link  Loss.  The  links  simulated  going  down 
are  the  links  connecting  Albany  -  Ft.  De trick,  Albany  - 
Andrews,  Gentile  -  Ft.  De trick,  Andrews  -  Hancock,  Andrews  - 
Ft.  Detrick,  Andrews  -  Tinker,  Ft.  Detrick  -  Tinker,  and 
Gentile  -  Tinker.  Appendix  H  contains  the  graphical  and  tab- 
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ular  results  of  the  eight  examples  of  link  losses.  They 
are  arranged  in  a  similar  fashion  as  those  given  in  Appen¬ 
dix  G. 

For  an  example  of  the  link  loss  sample  runs  (Figures 
H-13  through  H-15)*  the  Andrews  -  Tinker  link  is  chosen  for 
discussion.  The  impairment  to  the  network  as  a  result  of 
the  lost  link  is  evident  in  the  graphs.  In  both  the  all¬ 
packet  delay  and  the  short  packet  delay  graphs  (Figures  H-13 
and  H-14  respectively),  the  disturbance  period  for  the  60, 

70,  and  80  percent  saturation  runs  is  seven  seconds  (from 
time  frame  31  to  time  frame  38),  but  for  the  90%  run  it  is 
nine  seconds.  In  the  long  packet  delay  graph  (Figure  H-15), 
the  disturbance  period  is  six  seconds  for  the  60,  70,  and  80 
percent  runs  and  nine  seconds  for  the  90  percent  run.  The 
longer  recovery  times  indicate  a  threat  to  the  network's 
stability  as  the  result  of  the  loss  of  a  primary  link.  Al¬ 
though  well-behaved  in  the  60,  70,  and  80  percent  ranges, 
both  the  mean  and  maximum  packet  delays  exceed  those  of  a 
well-behaved  algorithm  for  the  90  percent  saturation  level. 

When  the  link  comes  back  up  (time  frame  42),  its  result  on 
the  network  causes  a  slight  smoothing  effect  on  the  packet 
delays. 

Sample  Node  Loss.  The  nodes  selected  for  removal 
from  the  network  sure  Ft.  De trick,  Gentile,  and  Tinker.  Appen¬ 
dix  I,  in  the  same  format  as  Appendices  G  and  H,  contain  the 
graphical  and  tabuleur  delay  results  for  the  node  loss  examples. 

The  removal  of  Ft.  Detrick  from  the  network  is  the  exam- 
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pie  from  the  set  of  node  losses  (Figures  1-1  through  1-3* 
Appendix  1),  to  be  discussed.  The  example  shows  the  disast¬ 
rous  effects  the  removal  of  a  primary  node  has  on  the  net¬ 
work.  On  all  three  graphs  the  disturbance  period  is  twelve 
seconds  (one  second  after  the  node  comes  back  up  at  time 
frame  42).  The  occurrence  of  any  moderate  traffic  spike 
within  the  twelve  second  time  period  would  most  likely  re¬ 
sult  in  massive  packet  delays  and  a  strong  possibility  of 
unstable  and  unbounded  output  trunk  queues  at  the  other  net¬ 
work  nodes.  The  algorithm  is  not  well-behaved  in  any  of  the 
packet  delay  graphs,  in  fact,  packets  incur  delays  of  up  to 
four  and  a  half  seconds  in  the  90  percent  run. 

Due  to  the  limited  number  of  priority  packets  gener¬ 
ated  (196  of  all  packets),  the  tabular  results  of  the  priority 
packet  delays  in  each  set  of  examples  discussed  above  does 
not  provide  conclusive  evidence  as  to  the  performance  of  the 
algorithm  for  priority  packets.  Inspection  of  the  results 
does  show  three  things  however 1  all  of  the  mean  and  maximum 
delay  values  for  short  packets  are  within  DCA  requirements, 
about  one  and  one  half  percent  of  the  long  packet  mean  de¬ 
lays  exceed  the  DCA  requirement  of  330  msec,  but  none  exceed 
the  maximum  delay  requirement  of  730  msec,  and  the  topological 
changes  seems  to  have  no  effect  on  the  mean  and  maximum  pac¬ 
ket  delays  for  priority  packets. 

Summary  Tables 

(  In  order  to  consolidate  and  present  all  of  the  infor- 
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mation  in  Appendices  F  through  I,  in  a  more  visible  manner, 
this  last  section  presents  four  tables  to  the  reader.  The 
thrusts  of  the  four  tables  are  to  show  the  adaptability  of 
the  hierarchical  routing  algorithm  through  the  two  perfor¬ 
mance  criteria  defined  previously  and  to  back  up  the  statis¬ 
tical  results  of  the  simulation  runs.  The  statistical  cal¬ 
culations  provide  95#  Confidence  Intervals  and  One-Tail  Null 
Hypothesis  tests  on  the  data  contained  in  Appendices  F 
through  I  on  which  the  four  tables  are  based. 

Using  a  95%  Confidence  Interval  to  determine  when  the 
network  returned  to  equilibrium  after  the  traffic  intensity 
spike,  Table  XI  gives  a  summary  of  the  various  disturbance 
periods  (in  seconds)  for  each  simulation  run  conducted. 

The  "All  -  Packets"  graphs  of  each  set  of  runs  were  used  to 
generate  Table  XI. 

Tables  XII  and  XIII  are  produced  using  the  Null  Hypothe¬ 
sis  Test  and  testing  for  m  at  «<  =  .05  for  non-priority 
short  packets  and  non-priority  long  packets  respectively. 

The  Null  Hypothesis  was  /*.  (**«•  =  380  msec  for  short  pac¬ 

kets,  =>  530  msec  for  long  packets),  and  the  Alternate 
Hypothesis  was  ju  >  m. •  The  tests  were  conducted  either 
assuming  large  sample  populations  with  a  normal  distribution 
or  known  standard  deviations  from  the  output  of  each  computer 
simulation  run.  The  "dashed"  entries  correspond  to  where 
the  Null  Hypothesis  was  accepted  and  the  "OVER"  entries 
correspond  to  the  rejection  of  the  Null  Hypothesis  in  favor 
of  the  Alternate  Hypothesis. 
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TABLE  XII  (CONT’D) 
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Gentile  -  Tinker 


TABLE  XIII  (CONT'D) 


The  last  table,  Table  XIV,  gives  a  summary  of  when  the 
algorithm  was  not  well-behaved  as  defined  by  the  author  pre¬ 
viously,  To  restate,  if  a  packet  has  a  mean  delay  greater 
than  or  equal  to  570  msec  for  short  packets  and  795  msec  for 
long  packets  or  has  a  maximum  delay  value  greater  than  or 
equal  to  2025  msec  for  short  packets  and  2250  msec  for  long 
packets,  then  the  algorithm  is  said  to  be  not  well-behaved 
at  the  traffic  level  in  which  the  excess  delay  occurred. 

The  "dashed"  entries  represent  when  the  algorithm  was  well- 
behaved  and  the  "X"  entries  correspond  to  when  the  algorithm 
was  not  well-behaved. 

The  last  chapter  presents  the  conclusions  and  recom¬ 
mendations  of  this  paper. 
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Algorithm  Behavioral  Ability 
(non-priority,  short/long  paokets) 


Ft.  De trick  -  Tinker 
Gentile  -  Tinker 
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IX  Conclusions/Recommendations 


The  purpose  behind  this  paper  was  to  determine  the 
adaptability  characteristics  of  the  hierarchical  routing 
algorithm  developed  by  System  Control,  Incorporated  for 
the  AUTODIN  II  computer  communications  network.  A  computer 
simulation  program  was  written  to  model  the  network  with 
the  routing  algorithm  written  in  Fortran  and  incorporated 
into  the  simulation  program. 

Two  network  parameters  were  varied,  traffic  volume 
through  the  network  and  network  topology  in  an  effort  to 
cause  the  routing  algorithm  to  adapt  to  these  changing  net¬ 
work  conditions.  By  taking  picture  "snaps"  of  the  network 
status  every  second  after  a  traffic  spike  occurred,  a  gener¬ 
al  idea  was  formed  as  to  how  the  algorithm  was  adapting. 

The  output  statistics  obtained  during  these  "snaps"  were  con¬ 
solidated  in  the  form  of  graphs  and  tables  in  Appendices  F 
through  I. 

The  previous  chapter  presented  a  summary  of  these  graphs 
using  accepted  statistical  procedures  in  order  to  help  the 
reader  form  some  opinion  on  the  adaptability  of  the  routing 
algorithm. 

Having  completed  the  simulation  exercise  the  author  has 
formed  these  conclusions i 

(1)  under  moderate  load  and  no  loss  of  primary  fac¬ 
ilities,  the  algorithm  adapts  very  well  and  does  not  cause 
undue  packet  delay. 

(2)  the  occurrence  of  traffic  spikes  and  network  top- 


ology  changes  has  little  or  no  effect  on  the  delay  incurred 
by  priority  packets  regardless  of  their  size. 

(3)  under  moderate  load  and  no  loss  of  primary  facili¬ 
ties,  the  disturbance  period  is  from  four  to  five  seconds. 

(4)  the  algorithm  is  well-behaved  during  the  majority 
of  simulation  runs,  but  in  some  cases,  does  cause  packets  to 
incur  delays  of  up  to  (and  exceeding),  six  and  one  half  sec¬ 
onds. 

Recommendations 

Throughout  the  simulation  exercise  there  were  two  links 
that  were  always  first  to  get  congested.  The  Albany  -  And¬ 
rews  link  and  the  Andrews  -  Hancock  link  only  contain  one 
trunk  each.  It  is  this  fact  plus  the  amount  of  traffic 
volume  they  carry  that  causes  the  link  to  become  congested. 

It  is  recommended  that  one  additional  trunk  be  allocated  to 
those  two  links. 

If  the  additional  software  overhead  would  not  be  too 
costly,  one  change  is  recommended  in  the  area  of  the  algo¬ 
rithm  that  builds  the  trunking  routing  table,  TTABL2.  In 
addition  to  evenly  distributing  the  number  of  packets  queued 
for  output  on  multi-trunk  links,  it  is  recommended  that  the 
distribution  of  packets  based  on  packet  length  also  be  used, 
line*  tiae  of  transmission  is  directly  proportional  to  the 
of  a  packet,  this  additional  qualification  on  packet 
•  •ration  to  the  trunks  could  possibly  reduce  packet  wait- 
m*  . n  output  queues  and  thus  reduce  the  overall  delay 
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time  • 

The  last  recommendation  is  one  of  individual  imple¬ 
mentation  of  the  algorithm  subroutines  into  the  simulation 
model •  Instead  of  having  64  subroutines  (eight  copies  of 
the  algorithm)  to  represent  the  algorithm  at  the  eight 
nodes,  the  implementation  should  be  changed  to  have  only  one 
copy  of  the  algorithm  but  accessible  by  the  eight  nodes.  If 
this  could  be  accomplished,  then  when  additional  nodes  were 
added  to  the  network,  they  too  could  access  the  single  algo¬ 
rithm  copy.  Otherwise  for  each  new  node  added  to  the  net¬ 
work,  eight  subroutines  would  have  to  be  written  for  that 
node's  copy  of  the  hierarchical  routing  algorithm. 
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Appendix  B 

Sample  of  Hierarchical  Algorithm 
Coded  in  FORTRAN 

Example  of 

Ft.  Detrick's  Copy  of  the  Algorithm 
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Appendix  C 

Albany's  Definitions  of  Subroutine  Table, 
Array,  Constant  and  Variable  Names 
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ALLD 


Link  Distance  (LD)  table  (Real),  network  topology 
matrix 

ALD  -  D  table  (Real),  contains  hop/congestion  information 

ALP  -  P  array  (Integer),  used  as  pointers  in  the  NC  array 

i 

ALNC  -  NC  array  (Integer),  list  of  the  connected  node  IDs 
to  a  particular  node  indexed  by  the  entries  in  the 
ALP  array 

ALROUT  -  ROUTE  array  (Integer),  identifies  the  link  to  take 
if  Albany  is  used  as  a  tandem  node 

ALSROT  -  SROUTE  array  (Integer),  identifies  the  link  to 
take  if  Albany  is  the  source  node 

ALHOP  -  HOP  array  (Integer),  number  of  hops  required  to 
reach  each  of  the  other  nodes 

ALLIST  -  LIST  array  (Integer),  list  of  connected  node  IDs 

ALLINS  -  array  containing  the  number  of  trunks  each  link  has 

ALBUFF  -  array  containing  the  number  of  packets  queued  for 
output  on  each  link 

ALBUFT  -  array  containing  the  number  of  packets  queued  for 
output  on  each  trunk 

ALBFID  -  array  containing  the  ID  number  of  the  first  trunk 
in  each  link 

ALLOCL  -  array  containing  the  results  of  the  calculation 
ALBUFF/ALLINS  for  each  link 

ALPARM  -  array  containing  the  six  parameters  associated  with 
each  packet  (from  GPSS) 

ALBUP  -  array  containing  the  current  status  of  each  link, 

1  -  connected,  1000  -  not  connected 

ALSTUS  -  array  containing  the  previous  status  of  each  link, 

1  -  connected,  1000  -  not  connected  j 

SAVVAL  -  array  containing  the  SAVEVALUEs  passed  to  Fortran 

subroutines  from  GPSS  I 

j 

INFI  -  pseudo  infinity,  1000  j 

ALNN  -  number  of  nodes  in  network,  8 

ALSELF  -  node  ID  for  Albany,  1 
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ALTBUF  - 
ALNLN  - 
ALS 
ALE 

ALZONE  - 

ALT1 

ALT2 

ALBNOM  - 

ALCNOM  - 
ALBEST  - 

ALSBES  - 

ALTEST  - 

ALSTES  - 

IDSRC  - 

IDSTN  - 

ALNODE  - 


total  number  of  packets  queued  for  input  processing 
number  of  link 3,  4 

variable  pointing  to  first  entry  in  the  LIST  array 

variable  pointing  to  last  +  1  entry  in  LIST  array 

node  saturation  flag,  1  -  saturated,  0  -  not  sat¬ 
urated 

link  congestion  threshold,  ? 
node  saturation  threshold,  1? 

congestion  bias  used  in  calculations  for  link  se¬ 
lection  entries  in  ROUTE  and  SROUTE  arrays,  ? 

extra  hop  bias  used  in  similar  calculations,  7 

variables  used  in  calculations  to  determine  the 
entries  in  the  ROUTE  and  SROUTE  arrays 

variable  used  in  calculations  to  determine  the 
entries  in  the  ROUTE  and  SROUTE  arrays 

variable  used  in  calculations  to  determine  the 
entries  in  the  ROUTE  and  SROUTE  arrays 

variables  used  in  calculations  to  determine  the 
entries  in  the  ROUTE  and  SROUTE  arrays 

source  node  ID  of  packet  (taken  from  parameter  4 
of  the  packet  -  ALPARM  (4) 

destination  node  ID  of  packet  (taken  from  para¬ 
meter  1  of  the  packet  -  ALPARM  (1)) 

index  value  into  SAVEVALUEs  pointing  to  the  first 
SAVEVALUE  corresponding  to  the  set  of  SAVEVALUEs 
belonging  to  the  source  node  of  a  status  packet 
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Appendix  D 


Full  Network  Simulation  Program 
Case i  Ft.  Detrick  Goes  Down 
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Appendix  E 

Listing  of  SAVEVALUEs 

I 

i 

I 

I 


Albam 


SAVE VALUE 


57  -  64 


Length  of  output  queue 


Reason  for  Use 
ALQ11 


"  "  "  "  ALQ21 

"  "  ■  “  ALQ22 

"  "  "  ■  ALQ23 

"  ■  "  "  ALQ31 

“  "  "  "  ALQ32 

"  "  M  "  ALQ41 

Network  status  information  (Albany's  row 
of  the  LD  matrix) 

Link  change  flag  if  non-zero,  trunk  change 
if  zero 

MESGEN  entry  flag,  0  -  STICK,  1  -  PTICK 


Length  of  input  queue 


ALQU1 

ALQU2 


MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  msg  generated 

Trunk/Link  change  flag,  0  -  drop,  1  -  add 

Link  number  in  which  change  is  to  occur 
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Andrews 


SAVEVALUE 

Reason  for  Use 

8 

Length  of  output  queue 

ANQ11 

9 

ft  ft  ft  ft 

ANQ21 

10 

NUN  ft 

ANQ22 

11 

ft  ft  ft  ft 

ANQ31 

12 

ft  ft  ft  ft 

ANQ41 

13 

ft  ft  ft  It 

ANQ42 

65  -  72 

Network  status  information 
of  LD  matrix 

(Andrews'  row 

130 

Link  change  flag  if  non-zero,  trunk  change 
flag  if  zero 

131 

MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

132 

Length  of  input  queue 

ANQU1 

133 

ft  ft  ft  ft 

ANQU2 

134 

MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  msg  generation 

135 

Trunk/Link  change  flag,  0  - 

drop,  1  -  add 

136 

Link  number  in  which  change 

is  to  occur 

Ft.  Detriclc 

SAVEVALUE  Reason  for  Use 


14 

Length 

of 

output  queue 

DEQ11 

15 

•f 

M 

99 

99 

DEQ12 

16 

m 

M 

M 

99 

DEQ13 

17 

99 

M 

M 

19 

DEQ21 

18 

99 

99 

ft 

H 

DEQ22 

19 

II 

H 

II 

99 

DEQ31 

20 

ft 

ft 

n 

99 

DEQ32 

21 

II 

H 

•• 

99 

DEQ33 

22 

N 

It 

it 

« 

DEQ41 

23 

n 

11 

n 

•1 

DEQ42 

24 

n 

II 

M 

•1 

DEQ51 

25 

H 

N 

M 

99 

DEQ52 

-  80 

Network 

status 

information  (Ft. 

De trick 

row  of  the  LD  matrix) 


140  Link  change  flag  if  non-zero,  trunk  change 
if  zero 

141  MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

142  Length  of  input  queue  DEQU1 

143  "  DEQU2 

144  "  "  "  DEQU3 

145  MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  message  generated 

146  Trunk/Link  change  flag,  0  -  drop,  1  -  add 

14?  Link  number  in  which  change  is  to  occur 


260 


Gentile 


SAVEVALUE 

26 

27 

28 

29 

30 

31 

32 

33 

34 

81  -  88 
150 


Reason  for  Use 


Length  of  output  queue 


GEQ11 

GEQ12 

GEQ13 

GEQ21 

GEQ31 

GEQ32 

GEQ41 

GEQ51 

GEQ52 


Network  status  information  (Gentile’s 
row  of  the  LD  matrix) 

Link  change  flag  if  non-zero,  trunk  change 
if  zero 


151  MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

152  Length  of  input  queue  GEQU1 

153  "  H  GEQU2 

15^  MESGEN  status  generation  flag,  0  -  no  msg, 

1  -  msg  generated 

155  Trunk/Link  change  flag,  0  -  drop,  1  -  add 

156  Link  number  in  which  change  is  to  occur 


Hancock 


SAVEVALUE  Reason  for  Use 

35  Length  of  output  queue  HNQ11 

36  "  "  HNQ21 

3?  "  M  HNQ22 

38  "  -  HNQ31 

89  -  96  Network  status  information  (Hancock's  row 
of  the  LD  matrix) 

160  Link  change  flag  if  non-zero,  trunk  chnage 
if  zero 

161  MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

162  Length  of  input  queue  HNQUE 

163  MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  msg  generated 

164  Trunk/Link  change  flag,  0  -  drop,  1  -  add 

165  Link  number  in  which  change  is  to  occur 
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McClellan 


SAVEVALUE 


Reason  for  Use 


39 

40 

41 

42 

43 

97  -  104 
170 

171 

172 

173 

174 

175 

176 


Length 

of  output  queue 

MCQ11 

M 

M  N  II 

MCQ12 

M 

•t  (1  M 

MCQ21 

N 

II  II  M 

MCQ31 

H 

n  If  II 

MCQ32 

Network  status  information  (McClellan's 
row  of  the  LD  matrix) 

Link  change  flag  if  non-zero,  trunk  change 
if  zero 

MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

Length  of  input  queue  MCQU1 

"  *  "  M CQU2 

MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  msg  generated 

Trunk/Link  change  flag,  0  -  drop,  1  -  add 

Link  number  in  which  change  is  to  occur 


( 
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Norton 

SAVEVALUE  Reason  for  Use 

44  Length  of  output  queue  N0Q11 

45  "  "  "  N0Q12 

46  ******  •*  N0Q21 

47  *'  "  N0Q31 

105  -  112  Network  status  message  information  (Norton's 
row  of  the  LD  matrix) 

180  Link  change  flag  if  non-zero,  trunk  change 
if  zero 


181  MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

182  Length  of  input  queue  NOQUE 

183  MESGEN  status  message  generation  flag, 

0  -  no  msg,  1  -  msg  generated 

184  Trunk/Link  change  flag,  0  -  drop,  1  -  add 

185  Link  number  in  which  change  is  to  occur 
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Tinker 

SAVE VALUE 

48 

49 

50 

51 

52 

53 

54 

55 

56 

113  -  120 

190 

191 

192 

193 

194 

195 

196 
197 


Reason  for  Use 


Length  of  output  queue 


TKQ11 

TKQ21 

TKQ22 

TKQ31 

TKQ32 

TKQ41 

TKQ42 

TKQ51 

TKQ52 


Network  status  information  (Tinker's  row 
of  the  LD  matrix) 

Link  change  flag  if  non-zero,  trunk  change 
if  zero 


MESGEN  entry  flag,  0  -  STICK,  1  -  FTICK 

Length  of  input  queue  TKQU1 

"  "  "  "  TKQU2 

nun  h  TKQU3 

MESGEN  status  message  generation  flag, 

0  -  no  rasg,  1  -  msg  generated 

Trunk/Link  change  flag,  0  -  drop,  1  -  add 

Link  number  in  which  change  is  to  occur 
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Appendix  F 

Graphical  and  Tabular  Delay  Results 
From  the  Fully  Operational  Network 
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SIMULATION 
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System  Delay  (priority  -  long  packets) 
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Appendix  Q 

Graphical  and  Tabular  Delay  Results 
From  the  Trunk  Loss  Runs 
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Albany  -  Ft.  De trick  Trunk  Loss 
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PACKET  DELAY 
700  (MSEC) 


Pig  G-5.  SYSTEM  DELAY  (NON-PRIORITY  -  SHORT  PACKETS) 


(MSEC) 


(SPIKE)  SIMULATION  TIME  (SECONDS) 


TABLE  G-III 
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TABLE  G-V 

System  Delay  (priority  -  short  packets) 

Simulation  Delay  Times  (milliseconds) 

Times  60%  Sat.  Run  70%  Sat.  Run  80%  Sat.  Run  90%  Sat 
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13.  SYSTEM  DELAY  (NON-PRIORITY  -  LONG  PACKETS) 


TABLE  G-VII 

System  Delay  (priority  -  short  packets) 

Simulation  Delay  Times  (milliseconds) 

Time  60 %  Sat.  Run  ?0#  Sat.  Run  80#  Sat.  Run  90%  Sat 
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TABLE  VIII 

System  Delay  (priority  -  long  packets) 

Simulation  Delay  Times  (milliseconds) 
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Appendix  H 

Graphical  and  Tabular  Delay  Results 
From  the  Link  Loss  Runs 
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Simulation  Delay  Times  (milliseconds) 
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